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Executive Summary 

This evaluation of progress in tackling the problems of access, quality, equity and completion of 
primary and secondary education examines the results of an alliance between the Coffee Growers 
Committee of Caldas (CGC) and the Department of Caldas, Colombia. The evaluation team 
employed a retrospective approach to understanding the social, economic, and educational 
contributions of the Alliance's programs serving the families of rural coffee growers. This report is 
intended to help inform ongoing decision-making within the programs. The design of the 
evaluation focused on implementation of the Escueta Nueva model at the local level, and on 
identifying the most important factors and conditions for successful local implementation. 

Questions central to this outcomes evaluation: 

■ To what extent financial investments in the Alliance programs led to significant economic, 
social, and cultural changes in the communities the programs served? 

■ How cost-effective the programs are based on student learning outcomes compared to 
the national average? 

■ What is the relationship between education and productivity in the Caldas region? 

■ To what extent do students demonstrate democratic and leadership skills? To what 
extent are students civically engaged? 

■ What institutional opportunities or challenges help or hinder students from succeeding in 
the workforce both during and after the program? 

In addition, the study examined the following elements: 

■ The extent to which access and quality objectives have been met; 

■ The implications of increased access, completion and learning; 

■ Specific desired learning outcomes; 

■ Students' demonstrated democratic behaviors. 

Methodology 

The retrospective approach uses participant's recollections to understand how systems change as 
a result of interventions. The AED team worked with the Centro de Estudios Regionales Cafeteros 
y Empresariales (CRECE) to collect data from a sample of 30 schools in the Caldas region. Ninety- 
six teachers and principals were interviewed using the CIT for grade 9. The Democratic Leadership 
Instrument, and a survey of the Alliance's Programs were used to collect data on perceptions of 
Grade 9 leadership skills. The data was triangulated with similar peer and teacher instruments in a 
360 degree review process. 

The Democratic Leadership Survey was completed by 267 students in grade 9, and by 30 grade 9 
teachers. The AED team also conducted 11 focus groups, applying the systems institutional change 
rubric with a total of 96 teachers, school principals, parents, staff of the Secretariat of Education 
(SED), alumni of the Alliance programs, and one-on-one interviews with local authorities and top 
leaders at the CGC. 

The cost, cost-effectiveness analysis, and economic impact analyses were conducted using 
secondary data provided to AED by the Alliance programs. The economic impact analysis used a 
statistical software package called SPSS to analyze frequencies and correlations between the 
variables. Univariate regressions were also conducted to measure the significance of education on 
productivity. 
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Findings 

The following are the major findings for each of the main components of this analysis. 

Cost and Cost-Effectiveness 

■ The Alliance programs were 4 times more cost-effective than the national average and 
nearly two times more cost-effective than the urban schools in Caldas. 

■ In terms of survival to grades 5, the Alliance programs were nearly two times more cost- 
effective at getting students to survive to grade than the national average and slightly 
more cost-effective than the schools in urban Caldas (US$2,049 vs. US$2,698). 

■ The Alliance schools were more than US$1,000 more cost-effective than rural schools in 
the Quindio and Risaralda regions. 

■ The Alliance schools were nearly two times more cost-effective than the national average 
with regard to grade 9, and also more cost-effective than similar coffee growing regions in 
Colombia. 

■ When the analysis compared the cost of obtaining learning outcomes, the Alliance 
programs continued to be significantly more cost-effective at helping students learn than 
the national average and surrounding regions. 

Economic Analysis 

■ Findings from the evaluation show that education correlates to increases in density of 
production (i.e. a proxy for productivity) at a significant level. 

■ The analysis further showed that coffee growers with a higher level of education have 
higher levels of production, so investing in education - particularly middle and secondary 
education contributes to economic development in Caldas 

Leadership and Democratic Behavior in Students 

■ Overall, students believed that their strongest skill was listening actively to others and 
hearing their words and feelings - a perception supported by the teachers. 

■ Peer groups felt that their peers were strongest in the areas of (a) Providing information 
about a subject at hand, and (b) teaching their colleagues to diagnose sources of 
difficulties. The student self-evaluations also gave these two categories high marks. 

■ Teachers, students and peers all agreed that students in the Alliance programs are 
effective at initiating ideas, actions and solutions - another strong characteristic of 
effective leaders. 

■ Areas identified as weaknesses and in need of improvement included: (a) Serving as a 
mediator; (b) eliciting information through open-ended questions, and (c) elaborating new 
ideas utilizing definitions and examples. 

Teacher Professional Development 

Teachers in the Alliance program felt the teacher training was one of the most critical elements in 

the Alliance programs, and they express a desire to receive more training throughout the year. 

They indicated that the training programs: 

■ Strengthened their skill set; 

■ Increased their ability to implement the pedagogical approaches; and 


Helped teachers implement innovative activities and different types of pedagogical 
practices. 
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Teachers raised concerns about the need for additional on-going training - particularly in smaller 
schools - and for a closer alignment of such training to preparing students for standardized 
testing. Twenty-nine teachers and school principals felt the pedagogical support visits were critical 
to successful teaching. 

The acceptance of the program's pedagogical practices by teachers was significant: forty-one 
teachers suggested that positive pedagogical practices were those in which students were active 
and entertained, and identified several in which learning was very effectively achieved. Some 
examples given were: 

■ Teaching Math with games - for instance, teaching geometry with puzzles; using bowling 
pins to add and subtract; 

■ Learning new vocabulary with the game Scrabble 

■ Learning the English words for body parts by touching them and saying the words at the 
same time 

■ Using computer programs such as Story Book to write stories. 

Teachers teaching in multi-grade settings faced more challenges than those in single-grade 
classrooms, including lack of time, increased need for general knowledge of various subjects, and 
more pedagogical interventions; higher teacher turnover seems to result from multi-grade 
settings. 

Additional Findings 

■ School staff and parents recognize the importance of the basic resources provided to the 
Alliance schools in providing quality education. 

■ Teachers and Principals raised concerns about some schools' lack of material resources 
such as sufficient classrooms and lab equipment for physics and chemistry coursework. 
Parents understand that since programs must provide students with practical experience, 
and provision of these resources is seen as critical. 

Learning guides were seen as the more important learning tool in the schools. However, 
respondents realize that the guides available for primary education are old and out of date. The 
Ministry of Education (MOE) has developed new guides, but the guides have not yet been 
distributed. Participants view getting the guides to the schools as critical to ensuring teacher 
preparedness and student learning. 

Having the Escuela Virtual program in rural schools with educational software programs and 
Internet access is seen as an advantageous tool that is used regularly within the schools. The labs 
are of great interest to children, and communities believe they prepare students for life after 
graduation. Teachers and students found the provision of computers and Internet access one of 
the most critical elements to student learning. 

In terms of student leadership, teachers, parents, and principals indicated that students were 
taking on various responsible roles such as being the table leader or materials keeper. The 
students were also in charge of their own learning since they are allowed to advance at their own 
pace. The self-paced nature of the curriculum allows for flexible promotion and supports students 
who need to be absent during the time of the harvest. 
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■ Teachers and principals have noted increased responsibility, civic engagement, and 
involvement on the farm as a direct result of the teaching methodology implemented in 
the Alliance programs. 

■ While implementing practical projects on the farm is a key aspect of learning in the 
Alliance programs, students did not view it as the most significant source of learning skills. 

■ The role of the facilitators is perceived as one of the most important contributions of the 
Alliance Programs along with the teacher circles and the teacher networks. 

■ Parents are demanding more information about what the students are learning, they want 
to be involved in the Alliance Programs. 

The Alliance programs have a strong, shared vision, strong leadership, and sufficient training 
provided to relevant actors. The institutional framework supports sustainability of the program. 
Issues such as the frequent and high teacher rotation, school consolidation, and the heavy 
administrative duties of school principals vs. instructional leadership are affecting the quality of 
the Programs and require a solution from the SED and MEN. 

Recommendations 

Internationally, many countries are looking for solutions to the expansion of secondary education. 
The Alliance has an effective "recipe" and they should make an effort to share their experience 
internationally. 

The following recommendations based on the findings of this evaluation are intended to support 
the continuation and improved quality of the Alliance's programs. 

1. Since the programs are more cost-effective in terms of access, survival and learning, 
communities, donors, and others should continue to invest in these. 

2. There is a positive correlation between increased levels of education and productivity. 

3. Focus on creating and developing incentives for students to stay in the coffee industry 
because it contributes to the economy of Caldas. 

4. Understand the causes and the effects that cause teacher turnover and implement 
strategies to ensure a more stable teaching force in the Alliance programs. 

5. Focus on improving the Learning Guides and ensuring the guides contain the most recent 
information. 

6. Understand why students do not find the individual projects as useful as other learning 
methods. 

7. Understand the most important data needs and collect that information. 

8. Ensure the permanence of the Alliance, based on legislation if possible. 

9. Increase public-private support for programs. 

10. Revise teacher training to include demand driven training and Link some aspect of the 
training to Standardized tests. 

11. Increase support to teachers in multi-grade classrooms. 

12. Improve coordination with the teacher preparation schools ( Normales ) and local 
universities. 

13. Enhance the role of school facilitators. 

14. Focus on improved student communication skills. 

15. Examine the possibility of SENA certifications for students under the age of 15 because it 
will act as an important incentive to "pull" students through the higher grades. 
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16. Develop school accountability tools in which parents can participate. 

17. Revise the school principal's emphasis on administrative duties vis-a-vis instructional 
support. 

18. Conduct additional research studies that contribute to strategic planning. 

19. Link teacher training to Standardized tests. 

20. The SABER tests only provide one perspective on learning and do not effectively measure 
the practical learning students undertake in the Alliance programs. The Alliance programs 
should look at alternative ways to measure learning, including evaluations of the student 
projects. 
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Introduction 

This evaluation was conducted for the Alliance between the Coffee Growers Committee of Caldas 
(CGC) and the Department of Caldas, Colombia (hereafter referred to as the Alliance). The 
evaluation employs a retrospective approach to understanding the social, economic, and 
educational contributions of the Alliance's Programs using the Escuela Nueva model and its 
associated programs in Posprimaria and Ensenanza Media, including Escuela Virtual; Escuela y 
Cafe; and Escuela y Seguridad AHmentaria. The evaluators used tools such as the systems 
institutional change rubric, the critical incidence interview technique (CIT), surveys, and document 
review to collect and analyze data. The Academy for Educational Development (AED) team used 
both quantitative and qualitative methods to examine how well the current programs support 
education quality as measured by learning outcomes. The results from the evaluation described in 
this report will serve to identify factors facilitating and impeding learning; conditions that allow 
the programs to be successful; as well as areas in which policy interventions will have the highest 
likelihood for improving instructional quality. The evaluation will help inform ongoing decision- 
making within the programs and provide useful guidance to the Alliance in their quality 
improvement efforts. 

The evaluation design focused on implementation of the Escuela Nueva model and methods at the 
local level, and on identifying the most important factors for local implementation and conditions 
for success. This focus is informed by evidence suggesting that understanding the technical, 
institutional, and political nature of reform efforts is critical to building an education system that 
has educational, social, and economic contributions. AED examined how contextual factors such as 
institutional capacity and framework, vision, political will and civil society influence the technical 
interventions implemented in these programs. Due to the complexity of factors influencing 
education policy and practice, this evaluation drew on prior research and evaluation that 
employed multiple methods and approaches to data collection to observe how the programs have 
affected the Caldas region. 

The evaluators sought to determine the main contributions of the Alliance's programs by 
responding to the following questions: 

■ To what extent have investments in the Alliance programs led to significant economic, 
social, and cultural changes in the communities the programs have served? 

■ How cost-effective are the programs based on student learning outcomes compared to 
the national average? 

■ What is the relationship between education and productivity in the Caldas region? 

■ To what extent do students demonstrate democratic and leadership skills? To what extent 
are students civically engaged? 

■ What institutional opportunities and challenges help or hinder student success both 
during and after the program? 
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In addition to the questions listed above, the study examined: 

■ The extent to which access and quality objectives have been met 

■ Increased access, completion and learning and its implications 

■ Learning outcomes 

■ Democratic behaviors 

The evaluation provided a baseline of student perceptions of their democratic, leadership, and 
civic engagement skills, which will allow the Alliance program to measure changes in student 
perceptions over time. 

Team Composition 

The AED evaluation team members are Dr. Audrey Moore, Acting Director of the agency's 
Education Quality Improvement Program 2 (EQUIP2); Ana Florez, Education Specialist and Project 
Director for AED; and Eva Grajeda, Program Officer for AED. Erin Monahan, Elizabeth Adelman and 
Ania Chaluda supported the analysis of the interviews. Erin Monahan, Erik Lundgren and Gabriel 
Alegrett supported the editing and production of this report. 

Methodology 

The AED team conducted this evaluation following detailed steps through the following phases: 

1. Background assessment and review of the literature 

2. Development of the design 

3. Data collection and analysis 

Background Assessment and Review of the Literature 

AED reviewed more than 18 documents related to the CGC programs during the months of 
December 2009 thru February of 2010. These documents were evaluations of the Escueta Nueva 
model, the implementation of Posprimaria and Education Media and some evaluations of 
particular associated programs. Many of the source document processes and qualitative changes 
show the depth of CGC commitment towards education in Caldas. Others were 
"sistematizaciones" such as the 25 years of implementation of the Escueta Nueva program by the 
CGC. 

In reviewing the existing CRECE documents and key evaluations that make up the international 
literature, the AED team found that there is considerable qualitative and perceptive data for all of 
the programs that highlight the types of skills students believe they are learning and the 
contributions that various stakeholders believe that the program is having. However, little 
quantitative data was found during the document review. It is important to have baseline data 
showing participant skill levels or learning levels at the entrance point of the program. This data 
was missing from the documents that were reviewed. 
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Development of the Evaluation Design 

The AED team used a combination of approaches to examine how the Alliance's programs attempt 
to improve education quality in schools serving Colombia's Caldas Region and make 
recommendations for modifications to improve effectiveness. The main approaches included: 

Quantitative Analysis: 1) Cost-effectiveness analysis against access, completion and learning, and 
2) Economic analysis of the relationship between coffee production and education. 

Qualitative Analysis: 1) The CIT of data collection that asks users to focus on one or more critical 
incidents, which they experienced personally in the field of activity being analyzed, 2) Democratic 
leadership skills, 3) The systems institutional change rubric as a retrospective approach that uses 
participant's recollections to understand how systems have changed as a result of interventions, 
and 4) Structured interviews of key actors. 

The AED team used the qualitative software package NVIVO to analyze the qualitative interview 
and focus group data. Qualitative research focused on exploring issues, understanding 
phenomena, and answering questions. Qualitative research uses unstructured information - field 
notes, videos, transcripts and audio recordings instead of numbers to arrive at conclusions. The 
NVIVO program allows researchers and evaluators to manage, shape, and make sense of 
information quickly and easily by creating information trees based on themes. 

The AED team developed a series of protocols for the data gathering process. These instruments 
were designed by adapting similar instruments used by AED to conduct comparable evaluations 
around the world. Instruments were developed for the systems institutional change rubric, the 
CTI, and the student leadership skills. 

The AED team identified a sample from the total number of rural schools in Caldas that receive 
Alliance support to collect the data. A sample size of 15% (30 schools) was necessary to yield valid 
statistical data from which inferences about the entire population could be drawn. The sample, 
selected randomly, represented schools of varying size and student enrollment rates. From 43 
central schools, 12 were selected (2 large, 6 medium, 4 small) 1 ; and from 148 non-central schools 
18 were selected (4 large, 6 medium and 8 small). 

The AED team worked with the CRECE team to collect all the data. Three separate teams were 
formed to visit each facility over the course of two weeks. Two of the teams had two researchers, 
while the last team had three. For 9 th grade, teachers and principals were interviewed using the 
CIT. Data on the perception of leadership skills for 9 th grade was collected by applying the 
Democratic Leadership Instrument, and a survey of the Alliance's Programs. The data was 
triangulated with similar peer and teacher instruments in a 360° 2 leadership review process. The 
three teams conducted 94 interviews, 76 of which were with teachers and 18 with either school 


1 Large schools refer to 300 or more students, medium from 151 to 299 students, and small from 62 to 150 students. 

2 A 360 survey is an assessment in which a person's peers respond to questions about their behavior. Those questions 
are also self-reported and the data is triangulated to look for alignment. The technique allows persons to see how their 
own perceptions of themselves compare to the ways in which others see them and their particular skill sets. 
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principals or coordinators. There were 267 ninth grade students and 30 ninth grade teachers who 
took the Democratic Leadership Survey. 

The AED team conducted 11 focus groups, applying the systems institutional change rubric with 96 
teachers, school principals, parents, staff of the SED, alumni of the Alliance programs, and one-on- 
one interviews with local authorities and top leaders at the CGC. 

The cost, cost effectiveness analysis and economic impact analysis was conducted using secondary 
data provided to AED by the Alliance programs. The Economic impact analysis used a statistical 
software package called SPSS to analyze correlations and frequencies of the variables. Univariate 
regressions were also conducted to determine the significance of education on productivity. The 
AED team proceeded to analyze and triangulate the data and to prepare this final report. 


Background 

Throughout the last two decades, Colombia has focused on improving its national educational 
system at the primary and secondary levels. Since the mid-1980s access and completion rates have 
increased dramatically, and numerous initiatives have been created to improve quality. In the span 
of eight years, the national net enrollment in primary education increased by 17 percentage 
points; from 77% in 1992 to 94% in 2000 (Figure 1). 


Figure 1. Colombia's net enrollment rate at the primary level (1985-2008) 
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Source: World Bank EdStats 2010 

For secondary education, the increase for the same time period was 15 percentage points; from 
40% to 58% (Figure 2). 
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Figure 2. Colombia's net enrollment rate at the secondary level (1992-2008) 



Source: World Bank EdStats 2010 


The net enrollment rate in 2008 for primary and secondary schooling was 90% and 71% 
respectively, and the 2007 survival rate 3 to grade 5 was 88% (World Bank, 2010; UNESCO 2010). 
While the Colombian government has made progress in providing access and increasing primary 
school completion, rural areas still remain a comparative challenge throughout the country. In 
2007, for every 100 rural school age children eligible to enroll in primary education, 88 students 
reached grade 5; and only 69 reached grade 9. In urban areas in 2006, for every 100 urban school 
age children, 92 survived to grade 5 and 71 survived to secondary education (UIS, 2007). 

Colombia's increases in access for primary and secondary students have resulted from a series of 
policies designed to improve access and quality on service delivery. One of the most important 
decisions made has been the allocation of additional financial resources to education resulting 
from an increase in the percentage of Colombia's GDP that is spent on education. During the 
1980s and part of the 1990s, the percentage of GDP that went into education was between 2% 
and 3%. However, for most of the 2000s, Colombia has allocated 4% of its GDP to education 
(World Bank, 2008). 

Furthermore, Colombia has aligned its education policies at the national level to make them 
consistent with the decentralization process at the departmental and municipal level. The 
participation of civil society in decision-making was one of the most important features of the 
education reform in Colombia. In that sense, Colombia has developed two participatory 
Educational Decennial Plans (1996-2005 and 2006-2015) in which civil society participated in 
establishing educational priorities for the country (World Bank, 2008). 


3 Survival rate is referred to the percentage of students enrolled in first grade in a given-school year who are expected to 
reach grade 5 and 9. It is a measure of internal efficiency of an education system. 
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In the last eight years, Colombia developed a National Development Plan "Revoiucion Educative/" 
that focused on improving the quality of education by using standardized tests results (pruebas 
SABER) to inform and improve teaching and learning. The Plan also set minimum learning 
standards in reading, mathematics, citizenship, and natural and social sciences. These examples 
demonstrate that Colombia has been investing in building a solid education system based on 
ensuring access to education not only in primary but also in secondary and tertiary schools, and by 
aligning policy frameworks that will improve learning in the long term. 

At the inception of these initiatives to improve education nationally in the 1980s, rates of access to 
education and school completion rates were low. In 1985, net enrollment for primary school was 
65% and the survival rate to grade 5 was 57% (World Bank, 2008). The situation was worse in rural 
areas and especially in regions where school-aged population density was low. One of these areas 
was the Department of Caldas, a coffee-producing region that was home to the Coffee Growers 
Committee of Caldas. Since 1981 the CGC has provided technical assistance and invested financial 
resources in many local development needs - including education. Investment was focused on 
school infrastructure in regions where a) coffee was grown, and b) at least 20 children were 
without access to school (CRECE, 2007). 

At the beginning of the 1980s, FICDUCAL and CGC 4 undertook some studies to determine the state 
of education in the Caldas region. The findings showed that the quality of education was poor, 
especially in rural areas; the dropout rate was high for those children who assisted in the coffee 
growing areas; schools lacked learning materials; absenteeism was high among teachers; and 
many children were malnourished. Generally, the studies indicated that education in rural areas in 
Caldas was failing to provide students with basic life skills, and that students dropped out early 
(CRECE 2007). 

The Alliance Supported Programs 

The Coffee Growers Committee of Caldas (CGC) was created in 1927 as part of the National Coffee 
Growers Federation. The Committee's long-term vision focuses on the development of a 
competitive and sustainable coffee growing industry that strengthens the social fabric in the 
Caldas region. To tackle problems of access, equity, dropouts, and quality of education in rural 
schools in Caldas, the CGC started investing in primary education in 1981 by adopting the Escuela 
Nueva methodology in rural schools. 

In an effort to build the human capital of Caldas in support of a strong and sustainable coffee 
industry, in 1986, the CGC entered into a partnership with the Departmental Government of 
Caldas (CRECE 2007). This partnership established that the CGC's support was to be technical while 
the government and SED contributions were to provide financial support and teachers (CRECE 
2007). Since its implementation, the partnership between the Departmental Government and the 
Alliance has expanded Escuela Nueva up to grade 11 and to include most of the rural schools in 
Caldas. The partnership has also added other complementary educational programs that meet the 

4 FICDUCAL study focused on the quality of education in rural areas. The CGC studies focused on supervisors and the 
Colonia Escotar program developed in rural areas. 
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needs of local communities. A short description off the main programs implemented by CGC is 
described in the following section. Conclusions from the evaluation related to each program are 
provided in the conclusion and recommendation section. 

Escuela Nueva 

The objectives of the Escuela Nueva initiative in Caldas by the CGC in 1981 were to strengthen 
rural education (grades 1-5) and provide an education that was more dynamic than that provided 
by the public school system. According to data available from the CGC, this program directly 
reached 1,113 schools in the Caldas Region, and has directly served an average of 5,000 students 
annually, and has trained approximately 3,200 teachers to improve their constructivist approaches 
(CRECE, 2007). 

The constructivist principles upon which the program is based include: 

■ Students are at the center of the learning process. 

■ Students set their own rhythm and pace for learning, with a curriculum that is self-paced 
and takes advantage of teamwork. 

■ Teachers maintain an open and positive environment and serve as facilitators for learning. 

■ The self-paced instruction is based on learning guides that contain activities and exercises 
that are sequenced to assist the learning process. The self-paced method of instruction 
promotes student initiative and creativity. The flexibility allows students to advance at 
one's own pace. 

■ Each school establishes a tight relationship between the community and school in which 
members of the family participate in the educational process and where community 
projects are a mainstay of the program. 

■ The student government employs strategies to ensure active participation of youth in the 
democratic life of the school, which strengthens values such as cooperation, respect, and 
teamwork. 

The Escuela Nueva has five main strategies that include: 1) teacher training through experiential 
workshops on the model, and specific trainings on core subject areas including Language and 
Math; 2) equipment and resources for schools including self-learning guides, teacher manuals, 
furniture, school libraries, among others; 3) frequent school visits by facilitators called padrinos to 
train teachers "on-the-job"; 4) small teacher networks ( microcentros ) in each school that meet 4 
times per year to discuss different components of the model, issues, and to share lessons learned 
and best practices; 5) Escuela Nueva teacher networks in each municipality, which were created to 
strengthen the Escuela Nueva model by reflecting on and evaluating the pedagogical components 
of this model. The networks support innovation among teachers on how to improve teaching and 
learning in their classrooms. 
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The Escuela Nueva model is the backbone of all Alliance programs. The CGC has expanded this 
model and created new programs that reach beyond primary education to the Posprimaria (6-9 
grades) and Education Media (10-11 grades), all using the same constructivist approaches. 

Posprimaria 

In 1989, Escuela Nueva began to expand the reach of education into secondary school. 
Posprimaria supports the continuation of the Escuela Nueva into grades 6 through 9. The program 
seeks to increase access to a relevant education and better use of educational and learning 
resources to support the development of human capital in the Caldas region. The program allows 
students to enhance skills that are in line with standard outputs from the MOE along with more 
general skills needed for effective transition into the workforce. 

A 1999 evaluation conducted by CRECE found that in terms of learning outcomes, Escuela Nueva 
and Posprimaria schools performed better than their public school counterparts. Generally, men 
performed better on the assessment in grade 7, but women outperformed the men in grade 9. 
Statistically, reading and writing scores for women were stronger than those of the men; both 
men and women performed poorly in mathematics. 

The study also presents results of the CAP national democratic competencies survey that 
measures democratic behavior. Results of the survey indicate that students have respect for 
others, are disciplined and have good manners; they value punctuality, attendance, and work in 
groups. The document includes data on efficiency and coverage, teacher profiles through desired 
characteristics in teachers, and data on school infrastructure. The study concludes by stating that 
80% of students are less than 20 years old and 56% continue studying upon completion of 
Posprimaria. The study also points out that the program does not appear to have reduced the 
number of those that eventually leave rural areas. 


Educacion Media 

High community demand and low coverage for grades 10 and 11 resulted in the Alliance 
establishing Education Media was established in 2004. At that time, as a result of a lack of schools 
offering grades 10 and 11, only 9.7% of students in the Department of Caldas who entered primary 
school completed grades 10 and 11. The success of the Posprimaria program encouraged the 
community to demand access to Education Media. The design of this program was based on the 
Escuela Nueva methodology and General Labor Competencies (GLC), an approach defined with the 
MOE. This planning was undertaken so that students who graduated from this program could be 
incorporated into the labor force. There were four competencies developed under the GLC: 
intellectual, relational, organizational, and personal. In order to start with the Education Media 
program, the partnership looked for additional local partners such as the Luker Foundation and 
Corpoeducacion. The program implemented new teacher training and developed teacher and 
student guides for both grades based on the Escuela Nueva methodology, GLC competencies, and 
the rural context. 
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Escuela y Cafe 

Established in 1996, Escuela y Cafe seeks to prepare students to become the next generation of 
coffee-growers in Caldas. From the outset, the program focuses on integrating knowledge of the 
coffee growing business into the curriculum and exposing students to an information process that 
teaches students the skills necessary to improve the production and management of coffee 
growing - an activity upon which their families are economically dependent. The program 
develops a series of modules and self-learning guides with practical activities for students to 
develop individual student projects (Proyectos Pedagogicos Productivos). The CGC works with the 
National Learning Service (SENA) to implement a diploma for students that show one of the seven 
norms and competencies needed for coffee production. The program outcome focuses on 
retaining students in the coffee-growing industry, but with improved skills that allow them to 
adopt better technologies, making production more efficient. 

A 2002 evaluation of Escuela y Cafe found that parents of students in this program believed that it 
related to the family's farm, better prepared students to enter the workforce, and improved the 
relationship between children and their parents by enabling them to work together on the farm to 
improve productivity. Students felt that the program was building useful life skills, and made such 
favorable changes as improving the inclusion of girls in activities outside the household - such as 
in the operation of coffee business functions - and in their status within the home. Students noted 
that as a result of the programs, they were gaining a voice in what happened on the farm and it 
increased their desire to stay and work in the coffee industry (CRECE, 2002). 

CRECE's 2002 evaluation of the partial impact and potential for Escuela y Cafe examined 60 
schools. The study uncovered that 83% of students found the materials useful compared to 33% of 
students in a similar program. Approximately 86% of students liked the pedagogical approach 
taken by teachers. Forty percent of students in the sample schools indicated that they were 
directly assisting in the management of the coffee growing industry and 3% were supporting the 
technical work involved in planting the coffee. 

Escuela Virtual 

Established in 1998, Escuela Virtual seeks to integrate technology and the information 
communication technologies (ICT) to support the learning process in basic education. The 
objective of the program is to ensure that students will develop ICT skills that allow them to 
integrate and adapt into the global knowledge economy. Existing impact evaluations show that the 
program has (a) created space for collaboration and integration of learning through the 
development of ICT projects, (b) changed the perception and value that is given to ICT as an 
educational delivery mechanism - particularly by teachers, and (c) strengthened relationships 
among public, private, and community organizations through the establishment of partnerships to 
assist student projects. A 2002 evaluation by CRECE further delineates teacher perceptions of this 
program and their growing support. 

Skills such as working together, using the internet to search for information, and using information 
to support learning are goals of the Escuela Virtual. 
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Escuela y Seguridad Alimentaria 

Established in 2005, Escuela y Seguridad Alimentaria focuses on the production and 
implementation of food security strategies. The program integrates the basic education curriculum 
with issues of food security to promote increased knowledge of nutrition among students. From 
2005-2006, the program began to implement the food security curriculum in 118 schools, 
targeting approximately 20,000 students. Approximately 700 teachers were trained during this 
time and 5,000 projects were developed and implemented related to food security issues. 


Contributions of the Alliance Programs: Access 

Alliance supported schools contributed to increased access to education in Caldas by addressing 
the major constraints to access, including the distance students have to travel, the economic roles 
children play in their families, the relevance of education, and the perceived relative value of 
formal education. As the table below demonstrates, enrollment in the Posprimaria programs rose 
from 7,952 students in 2002 to 11,823 students in 2007. Without the support of the Alliance 
programs, these students would likely have either dropped out of the system or left the rural area 
of Caldas to continue their education. This evaluation was not able to determine the reasons for 
the decrease in enrollment in the number of students enrolled in the Escuela Nueva. However, the 
decrease could be explained by policy changes that occurred at the national level when the 
government instituted its school consolidation policy. This policy moved students from one school 
to another and made it difficult to track exact enrollment numbers. Moreover, the National 
Coverage Information System (SIMAT) was not as well developed in 2002 so it is likely that the 
data from 2002 was less accurate. It is important that a future evaluation try to understand 
whether the decrease in enrollment is due to drop out, migration or data collection so that the 
Alliance programs can ensure a sustained and targeted enrollment. Many countries are moving, 
with different rates of success, from ensuring access in primary to secondary education. It is worth 
noting that one of the major accomplishments of the Alliance Programs has been increased access 
to secondary education (Posprimaria and Educacion Media) - as demonstrated in Table 1. The 
Alliance Programs are clearly having the "pull" effect of helping students to enroll and complete 
additional educational opportunities. 


Table 1. Enrollment Growth through the Alliance Programs 



Number of students in each 
program - 2002 

Number of students in each 
program - 2007 

Escuela Nueva 

37,590 

32,529 

Posprimaria (including Escuela y Cafe, 
Virtual, Alimentaria) 

7,952 

11,823 

Educacion Media 

n/a 

1,251 


Source: Data from CGC logic frameworks and SED, 2002 and 2007 

According to the Alliance, 678,215 students have participated in pre-primary to grade 11 from 
1997 to 2010. As Table 2 demonstrates, as the Alliance programs expanded access by adding 
additional grades, more students completed grades 1-6 and continued into middle and secondary 
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education. The number of students enrolling in grade 10 increased more than 9 times the initial 
enrollment of 120 in 2003 - increasing to 1,190 by 2010. The number of students entering grade 
11 more than tripled in the six-year period between 2004 and 2010. Based on this Table, the 
Alliance programs have clearly helped to increase access and completion in the rural regions of 
Caldas. 

Table 2. Rural Enrollment by Grade in Caldas 


RU RAL EN ROLLMENTBY G RADE (#OF STU DENTS) - CALDAS 


YEAR/GRADE 

P 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

TOTAL 

1997 

- 

11,323 

8,725 

7,534 

7,251 

6,875 

1,156 

869 

656 

625 



45,014 

1998 

- 

12,533 

9,254 

8,733 

7,986 

6,349 

1,689 

1,394 

919 

550 



49,407 

1999 

1,100 

10,953 

9,374 

8,855 

7,210 

6,129 

2,551 

2,121 

1,574 

954 



50,821 

2000 

1,150 

11,433 

9,276 

8,754 

6,790 

5,634 

2,886 

2,167 

1,636 

1,561 



51,287 

2001 

1,020 

12,646 

8,110 

8,028 

6,841 

5,727 

2,599 

2,206 

1,891 

1,658 



50,726 

2002 

2,225 

11,255 

7,965 

7,310 

5,850 

5,210 

2,782 

2,089 

1,577 

1,504 



47,767 

2003 

2,378 

9,827 

7,524 

6,947 

5,501 

4,909 

3,044 

2,201 

1,740 

1,654 

120 


45,845 

2004 

2,754 

10,013 

7,978 

7,515 

5,056 

4,818 

3,160 

2,373 

1,791 

1,709 

227 

151 

47,545 

2005 

3,104 

9,260 

7,236 

6,601 

6,234 

5,458 

3,557 

2,670 

2,016 

1,923 

419 

279 

48,757 

2006 

4,421 

8,564 

7,351 

6,895 

6,260 

3,459 

4,023 

3,021 

2,280 

2,217 

622 

415 

49,528 

2007 

4,375 

7,474 

7,150 

6,484 

5,945 

5,476 

4,137 

3,106 

2,344 

2,236 

813 

438 

49,978 

2008 

4,088 

6,427 

6,029 

5,636 

5,104 

4,694 

5,152 

3,349 

2,957 

2,260 

1,277 

750 

47,723 

2009 

4,063 

6,347 

5,826 

5,701 

5,468 

5,062 

3,990 

3,497 

2,951 

2,498 

1,160 

625 

47,188 

2010 

3,822 

5,982 

5,538 

5,329 

5,228 

4,989 

4,395 

3,677 

3,156 

2,582 

1,190 

741 

46,629 

TOTAL 

34,500 

134,037 

107,336 

100,322 

86,724 

74,789 

45,121 

34,740 

27,488 

23,931 

5,828 

3,399 

678,215 


Source: Departmental Coffee Growers Committee of Caldas (2010) 


Contributions of the Alliance Programs: Completion 

In terms of urban and rural rates in Caldas, for every 100 students, 81 students complete grade 5 
in urban schools compared to 61 in rural regions; 59 students complete grade 9 in urban areas 
compared to 27 in rural schools. These results are consistent with other countries in the LAC 
region. 

According to available information, drop out is generally not a major issue in the Alliance schools. 
Student attendance is high and, once enrolled, students continue on to grade 5 with impressive 
levels of commitment and enthusiasm, and the number has steadily improved. Teachers and 
parents contribute to the high rates of grade completion in the Alliance schools. Teachers tend to 
be present in the schools and parents ensure that students attend. Involvement by parents on the 
local school boards further contributes to ensuring that students are present in school and are 
completing grade levels. 

In terms of survival, the team analyzed data from the Alliance schools that had grades 1-9 
beginning in 2002 to see if survival rates would be higher in schools that offered education beyond 
grade 5 for a longer period of time. Since repetition rates are not available for Alliance programs, 
given the flexible promotion policy the program follows, the evaluators applied national repetition 
rates to a cohort analysis to calculate survival rates for students in the schools that had grades 1-9. 
Evaluators then conducted a sensitivity analysis looking at whether the survival rates would 
change if repetition were increased or decreased by 1,5, and 10 percentage points. Results 
indicate that in 2009, survival rates to grade 5 were approximately 72%. Survival rates from grade 
6 to grade 9 are higher at 75%. 
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It is important to understand that prior to 2000, few schools existed in rural Caldas that offered 
opportunities for students to continue their education beyond grade 5. As the previous section on 
access demonstrated, the Alliance programs have significantly increased access to education 
beyond grade 5 and so students are now moving through the higher grades in greater numbers, 
particularly when compared to national averages in Colombia. 


Contributions of the Alliance Programs: Learning 

The Government of Colombia conducts national testing of students at the grade 5 and grade 9 
levels in both math and language, called Pruebas SABER. Students from the Alliance also take 
these exams for comparable learning data, where available, for assessment. The tests are 
calculated and, based on the number of points received, are placed in one of four categories: 
insufficient (0-226 points), minimum (227-315 pts), satisfactory (316 - 399 points), and advanced 
(400-500 points). For the purpose of this analysis, this report has combined the outcomes for the 
satisfactory and advanced categories, and considers students who achieve the satisfactory level or 
better to have met the "learning threshold." 

For grade 5 language, the percentage of students at the national level reaching the satisfactory or 
advanced levels (316 or higher) was 33%. No average percent reaching the threshold was available 
for the Alliance programs, but the estimated average is similar to that of the Caldas region (58%), 
given the common cost/learning outcome and survival rates. Based on these estimates, the cost 
per learning in the subject of language for the Alliance programs is US$5,254 - slightly more cost- 
effective than the urban region of Caldas (US$6,131), and significantly more cost-effective than 
the national average of US$14,830. In grade 5 math, 30% of students were able to reach the 
satisfactory and advanced levels at a cost of US$6,831. The percentage of grade 9 students 
reaching the satisfactory and advanced levels was 20%. While the cost is higher per learning 
outcome than for completion and access, the Alliance programs are still more cost-effective than 
the national average as well as the averages for similar regions (Quindfo and Risaralda). 

It is important to note that standardized tests are the most common measure for learning and the 
only test available for the purposes of conducting the cost-effectiveness analysis. Since the 
Alliance programs teach students to design and implement more effective techniques for coffee 
production, it would be important to use other measures to compare learning in this context. For 
example, in Brazil, the PROJOVEM program conducted a cost-benefit analysis by using student 
projects as a measure of learning and resource contribution. Evaluators measured the returns on 
the investment of student projects compared to those of students from the traditional vocations 
schools. Results showed that the PROJOVEM programs were more cost-beneficial due to the 
business design of the projects and the productivity gained by increased student knowledge from 
the PROJOVEM program. It is important for the Alliance programs - particularly the Escueta y Cafe 
and Escuela Alimentaria to develop projects or tests that measure learning specific to the 
knowledge students are gaining because the SABER tests do not capture this learning contribution. 
For that reason, this evaluation examined additional variables such as student leadership behavior, 
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school observations, and qualitative data from students, teachers, school principals and parents to 
look at the broader contributions of the Alliance programs. 

It is also important to note that national and regional averages mask specific school-level 
variations. So while the average result for the rural areas seems to indicate that students in the 
Alliance programs performed similarly to other regions and the national average, many individual 
schools in the Alliance programs significantly outperformed the aggregate averages. Figures 3 and 
4 show the variation in schools. 

Figure 3. Student Performance in Rural Caldas exceeding National Average 


100 % 

90% 

80% 

<A 

70% 

<D 

■O 60% 

3 

Xf) 50% 
a? 40% 

30% 

20 % 

10% 

0% 

□ Satisfactory+Advanced ■Basic+Below Basic 



Source MOE (2010). Education Sector Statistics of Caldas 


As Figure 3 demonstrates, 30 of 117 education institutions in the rural region of Caldas exceed the 
national average in Grade 5 Language results. 

Figure 4. Student Performance in Grade 5 Mathematics exceeding National Average 



Source MOE (2010). Education Sector Statistics of Caldas 


In Math, 34 of the 117 educational institutions exceeded the national average. It is also important 
to note that in several of the municipalities of Caldas, the rural areas exceeded the urban area in 
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learning performance. As Figure 4 demonstrates, 11 of the 27 municipalities in Caldas 
outperformed students in the urban areas. 


Figure 5. Rural and Urban Performance on the SABER exams 



Source MOE (2010). Education Sector Statistics of Caldas 

In conclusion, while the average performance for students in rural and urban Caldas and at the 
national level are similar, one can see that in many cases, the variance among schools is great. 
Many of the Alliance schools outperformed the urban areas and national averages in both 
Language and Math test results. Moreover, it is important to remember that the SABER tests do 
not measure the direct knowledge, skill, and productivity contributions of these programs, which 
are discussed later in this report and are critical to the success and education contributions that 
the Alliance programs make to the coffee growing industry. 


International Comparisons of Access, Completion and Learning 

It is important to understand these findings in the context of other international programs serving 
underserved populations in rural regions. In 2006, the AED Educational Quality Improvement 
Program (EQUIP2) research team conducted a cost and cost-effectiveness analysis of 10 
complementary education programs and examined how these programs increased access, 
completion and learning in comparison to the traditional government systems. The following table 
provides a summary of the effectiveness of these programs. 
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Table 3. Complementary Education Program Effectiveness— Access, Completion and Learning 


Program 

Access 

Completion 

Learning 

Afgh: 

COPE 5 

In 6 provinces, provided 9% of the 
enrollment in 2003. In 2001, 
provide the only access for girls. 

COPE: 50% 
Public: 32% 

% passing end of 
year 
COPE: 94% 
Gov't: na 

Afgh: IRC 

Where located, provides the only 
access to education, especially for 
girls 

IRC: 68% 
Public: 32% 

% passing end of 
year 
IRC: 99% 
Gov't: na 

Bangladesh 

Accounts for 8% of national 
enrollment. May account for over 
50% of the total enrollment in 
rural areas 

BRAC: 94% 
Public: 67% 

% passing basic 
competencies 
BRAC: 70% 
Gov't: 27% 

Egypt 

Where located, provides the only 
access to education, especially for 
girls 

CS: 92% 
Public: 90% 

% passing 5 th grade 
exam 
CS: 94% 
Gov't: 73% 

Ghana 6 

Raises the enrollment rate for 
grades 

1-3 in Northern Region from 69% 
to 83% 

SFL: 91% 
Public: 59% 

% meeting minimum 
standards 
SFL: 81% 
Gov't: na 

Guatemala 

Accounts for 15% of national 
enrollment, 

PRONADE: 98% 
Public: 62% 

NA 

Honduras 7 

Over life of program, served about 
30% of out-of-school population 

Educatodos: 61% 
Public: 68% 

% passing basic 
competency 
Educatodos: 63% 
Gov't: 62% 

Mali 

Increased the enrollment rate in 
Sikasso from 35% to 62% 

CS: 67% 
Public: 56% 

% passing CEP exam 
CS: 51% 
Gov't: 43% 

Zambia 8 

Increases national enrollment by 
25%. As many as 30% of 
community school students are 
orphaned or vulnerable 

CS: 72% 
Public: 72% 

% meeting minimum 
standard 
CS: 40% 
Gov't: 35% 


Source: Table taken from DeStefano, Moore, Balwanz, and Hartwell (2006). 

As the Table above demonstrates, in Afghanistan, Bangladesh, Egypt, Ghana, Guatemala and Mali, 
the complementary education programs achieved completion rates that surpass those of the 
formal public schools in each country (DeStefano et al., 2006). Similar to the Alliance programs, 


5 In both programs in Afghanistan data is not available for regular public schools. The end of year exams 
reported for the complementary programs are teacher administered tests. While these are not good 
objective measures of student performance, they do provide some indication of how students are doing in 
those schools. 

6 For Ghana, data concern grades 1-3. 

7 For Honduras, the completion data are for grades 1-6, learning data are for grades 7-9. 

8 In Zambia, a national completion rate of 72% is reported, which includes both government and community 
schools. Since it was not possible to disaggregate the contributions of government and community schools, 
72% is used for both. 
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these programs increased access to primary and secondary education and assisted each of the 
countries in moving towards their EFA targets. Without the programs, many students would not 
have had access to education. 

In Bangladesh, Egypt, Honduras, Mali and Zambia, it was possible to compare learning outcomes 
of community and public schools using available data from a single measure for both categories of 
schools. In Bangladesh, Honduras and Zambia community and public school students sit for the 
same exams that measures learning in core subjects against an objective benchmark. In 
Bangladesh a much higher percentage of BRAC students than government students meet the 
benchmarks for basic competencies in all subjects - 70 percent compared to 27 percent. In 
Zambia, 40 percent of community school students met minimum standards in reading compared 
to 35 percent of government school students. In Mali and Egypt, pass rates on the end of primary 
cycle examination for community school students are higher than for regular public school 
students (DeStefano et al., 2006). 

It is important to note that these programs, with the exception of Educatodos, address access, 
completion and learning in primary school. Once students complete these programs, they must 
enter the regular government system if they want to continue their education. The Alliance 
programs provide students with the option of continuing study in the higher grades without being 
forced top to leave their rural locales. The access and completion rates for the Alliance programs 
in grades 9-11 are equivalent or higher than the rates for similar programs that provide only a 
primary level education. 


Cost and Cost-Effectiveness of the Alliance Programs 

When conducting a cost-effectiveness analysis, both the costs and effects of alternative programs 
are analyzed in relation to the production of a certain outcome. By conducting such an analysis, 
decisions on using the most cost-effective option(s) can divert resources to other areas of 
education. As Levin and McEwan state, "Cost-effectiveness analysis should be a topic of concern 
because it can lead to a more efficient use of educational resources - it can reduce the costs of 
reaching particular objectives, and it can expand what can be accomplished for any particular 
budget or other resources constraint" (Levin and McEwan 2001, p. 6). 

In this case, a cost and cost-effectiveness analysis was undertaken by comparing the cost- 
effectiveness of Alliance-supported programs to the cost-effectiveness of regular public schools at 
the national and departmental levels. It is important to note that all of the rural schools in the 
department of Caldas participate in the Alliance programs, and that all participating Alliance 
schools are government-run. As a consequence, the AED team compared cost and cost- 
effectiveness of results to two similar coffee-growing regions: Quindfo and Risaralda. The costs of 
access and learning were calculated and evaluated against outcomes on the SABER exams. 


The intention of this analysis was to compare the cost-effectiveness of the Alliance programs with 
other programs in Colombia and to demonstrate that the Alliance programs are an important 
investment, particularly for rural regions. 
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The cost of the Alliance programs was examined from various perspectives. First, the analysis 
looked at the cost in US dollars to run such programs and the sources of funding. Then, within that 
total cost, the analysis examined the cost structure and the major program operating costs. 

Table 4. Recurrent Costs of the Alliance Programs (2007) 


Programs 













Total 


Equipment 

% of 
Total 

Teacher 

Salary 

% of 
Total 

Training 

% of 
Total 

Technical 

Support 

% of 
Total 

Administratis & 
Operations 
Support 

% of 
Total 

Materials 

% of 
Total 


Escuela Nues 

64,181 

1% 

7,286,407 

97% 

38,226 

1% 

99,510 

1% 

1,287 

0% 

44,580 

1% 

$7,534,191 

Posprimaria 

152,902 

5% 

2,924,473 

92% 

40,236 

1% 

47,667 

2% 

2,981 

0% 

9,438 

0% 

$3,177,697 

Escuela y Caf 

r 91,081 

45% 

- 

0% 

37,376 

18% 

61,718 

30% 

1,494 

1% 

12,042 

6% 

$203,711 

Escuela Virtual 

26,731 

24% 


0% 

30,014 

27% 

48,798 

43% 

1,630 

1% 

5,216 

5% 

$112,390 

Escuela y 
Seguridad Alimen. 

174,611 

74% 

- 

0% 

28,315 

12% 

33,978 

14% 

- 

0% 

- 

0% 

$236,904 

Educacion Media 

91,553 

10% 

625,853 

69% 

41,199 

5% 

78,928 

9% 

1,630 

0% 

65,515 

7% 

$904,678 

Total 

$601,058 


$10,836,733 


$215,366 


$370,599 


$9,023 


$136,792 


$12,169,571 


Source: Original tabulation completed by authors based on information provided by The Alliance program staff. 


The breakdown of the total recurrent costs from 2007 for the Alliance programs is illustrated in 
Table 4. The recurrent expenditures include teacher salaries (69-97%); equipment (1-74%); 
training (1-27%); and materials (0-7%). As is typical in the public education system, teacher salaries 
comprise the majority of the recurrent budget - approximately 97%. 

According to the available information, Table 5 presents the funding sources for total 
development and recurrent costs for all programs. The Alliance is the main contributor to all 
programs except Escuela y Seguridad Alimentaria, which the Government of Caldas funded by 
itself in 2007. Microsoft and the Luker Foundation contributed only to the Escuela Virtual and 
Educacion Media programs respectively. The Government of Caldas is also responsible for funding 
all teacher salaries for each program (CGC and SED, 2007). 

Table 5. Funding Sources for Alliance Programs-Development and Recurrent Costs (2007) 


Entity 

Development 

Recurrent 

Total 


Contribution 

% of 
Total 

Contribution 

% of 
Total 


CGC 

118,117 

19.65% 

294,190 

2.54% 

$412,307 

Departmental Gov’t of Caldas 

438,768 

73.00% 

11,191,042 

96.74% 

$11,629,811 

Cooperative de Caficuitores de Manizales 

30,675 

5.10% 

21,503 

0.19% 

$52,178 

Microsoft 

12,318 

2.05% 

8,682 

0.08% 

$21,000 

Luker Foundation 

1,180 

0.20% 

53,095 

0.46% 

$54,275 







Total 

$601,058 

100.00% 

$11,568,513 

100.00% 

$12,169,571 


Source: Original tabulation created by authors 9 based on information provided by the Alliance 


The cost-effectiveness of the Alliance programs can be evaluated in terms of the average cost to 
produce a primary school survivor (student that passes up to grade 5 and then grade 9). 


9 Data from Microsoft and Luker Foundation was included to show complementary investments to those done by the 
CGC. 
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Table 6 captures the annual recurrent cost per pupil, 10 survival rates up to 5th and 9th grades, cost 
per completer, percentage of students who reached the desired outcomes, 11 and cost per learning 
outcome for the national, departmental, and Alliance programs. 

This analysis shows that the Alliance programs have been cost-effective at reaching underserved 
populations and ensuring that they survive up to grades 5 and 9 of the education system. For 
example, the average annual cost per student in the Alliance's primary education programs is 
US$250 compared to US$1,044 at the national level; US$437 in urban Caldas; and US$434 and 
US$437 respectively in Rural Quindfo and Risaralda. At the post-primary level (grades 6-10), 
Alliance programs cost US$274/student compared to US$838 at the national level and US$445 in 
rural Quindfo and Risaralda. 

The cost-effectiveness of each Alliance program can be evaluated in terms of its average cost to 
produce a primary school survivor. Survival rates for Alliance students in grades 1-5 averaged 
approximately 61% in 2007 compared to 88% percent at the national level. Despite the lower 
survival rates, the cost per survivor in Alliance programs is approximately US$2, 049/student 
compared to US$5, 932/student in government schools; US$2,698 in urban Caldas; and over 
US$3, 500/student in the rural areas of Quindfo and Risaralda. 12 The differences in cost per student 
stem partially from the fact that per student costs are lower for the Alliance program in rural 
Caldas. 

Similar results were found for grade 9, with the cost per survivor in grade 9 for the Alliance 
programs was US$5,309 compared to US$10,078 at the national level and over US$8,700 in the 
rural areas of Quindfo and Risaralda. Since survival rates for the Alliance programs were difficult to 
confirm, a sensitivity analysis of various survival rates (i.e. 85%, 95%, 100%) was conducted and 
comparisons drawn to the national and regional statistics. Survival rates in the Alliance programs 
would have to drop to below 30% for the cost-effectiveness to match the national level. 

In terms of the cost-effectiveness of learning outcomes, the Alliance programs were more cost- 
effective at reaching learning outcomes in both grades (5 and 9) as well as in both subjects. In 
grade 5 language, the cost of reaching learning was more cost effective at US$5,254, compared to 
the urban region of Caldas (US$6,131), and significantly more cost effective than the national 
average of US$14, 830/student. In grade 5 math, a less effective cost per learning outcome for the 
Alliance programs began to show lower percentages of students who were able to reach the 
satisfactory and advanced levels in math (US$6, 831/student). Score ranges for the Alliance schools 
were between 0 - 100%, with most values consistently under 20%. While the cost was higher per 
learning outcome, it was still more cost effective than the national average, and the averages for 
similar regions (Quindfo and Risaralda). 


10 Annual recurrent cost per-pupil for the Alliance's programs was calculated as an average of all the Alliance's programs. 
Cost information and data was provided by CGC. 

11 Number of students used to calculate per-pupil cost was taken from logic-frameworks provided by CGC. Percentages 

were taken from ICFES website. 

12 

The cost per student and survivor and learning rates in the regions of Quindio and Risaralda are estimates. The data 
was not available to calculate survival rates for these regions so we assumed that the regions had a similar pattern to 
Caldas and applied the same survival rates. 
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Table 6: Cost-Effectiveness of Alliance Programs 







Annual Per-Pupil Cost 





Primary 

1044* 

$250 A 

$434 

$437** 

Pos-primary 

838* 

$274 

$442 

$445** 

Survival Rate 





To 5th Grade 

88%* 

61%*** 

61% AA 

61% AA 

To 9th Grade 

69%§ 

27%*** 

27% AA 

27% AA 

Cost Per Completer 





To Grade 5 

$5,932 

$2,049 

$3,557 

$3,582 

To Grade 9 

$10,078 

$5,309 

$8,718 

$8,778 

Percentage of Students 
Meeting Learning Outcomes 





Reading 5th 

40% 

39% 

43% 

42% 

Reading 9th 

43% 

38% 

54% 

47% 

Math 5th 

33% 

30% 

31% 

32% 

Math 9th 

26% 

20% 

34% 

29% 

Cost Per Learning Outcome 





Reading 5th 

$14,830 

$5,254 

$8,273 

$8,528 

Reading 9th 

$23,437 

$13,972 

$16,145 

$18,676 

Math 5th 

$17,975 

$6,831 

$1 1 ,475 

$11,194 

Math 9th 

$38,761 

$26,546 

$25,642 

$30,268 


Source: Original tabulation completed by authors 

Notes: * The per-pupil cost of access for primary and secondary at the national level and completion rate to 5th grade were obtained 
from the UNESCO (2010) EFA report and represents costs for 2007 

** The per-pupil cost of access for primary and secondary at the Caldas departmental level is calculated with information provided by 
SED. Number of students for each grade level used to calculate per-pupil cost was taken from logic frameworks 2007 provided by CGC. 
*** Please see Annex 2 for explanation of how survival rates for the department of Caldas and CGC schools were calculated. 

A Annual recurrent cost per-pupil for CGC programs was calculated with information provided by the CGC, and represents cost for 
2007. 

^ Source: EPDC (2010). "Education Policy and Data Center, Sub-national Data." Washington, DC. Available at: http://www.epdc.org/ 
AA For the purpose of this analysis, survival rates used for 5th and 9th for Quindio and Risaralda were the same as the Caldas. This is due 
to the lack of data for both departments needed to calculate these rates. These departments have similar characteristics (coffee 
growers), and have similar survival rates 


For both grade 9 language and math, the Alliance programs had fewer students able to reach the 
satisfactory and/or advanced levels as previously indicated. Therefore, the programs were not as 
cost-effective per learning outcome, though still more cost-effective than the national averages 
(i.e. US$26,546 vs. US$38,761). This is another subject that teachers should place additional focus 
on when working with students to improve learning. 

One point, which is important to note is that the SABER test results reflect results for students in 
both the public and private education systems, but the national cost calculations do not include 
the costs of private education. As a result, it is likely that the true cost per student for the national 
average is higher than currently shown. 
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When the Alliance programs were compared with other similar complementary programs in other 
countries, the unit recurrent costs were lower per student - consistent with programs in 
Afghanistan, Guatemala, Egypt, Bangladesh, Honduras, and Zambia, compared to the regular 
public schools in the same country (DeStefano et al., 2006). 

Completion rates are higher in the complementary programs in both cases in Afghanistan, and in 
Bangladesh, Egypt, Ghana, Guatemala, and Mali - again, similar to the Alliance programs. In all 
cases except Mali, the costs per completer (or survivor) in complementary programs are lower 
than the cost per completer in regular public schools (DeStefano et al., 2006). 

In Bangladesh, Egypt, Ghana, Mali and Zambia, it was possible to compare learning outcomes for 
complementary and public schools. In each of those cases, the learning outcomes are higher in the 
complementary program schools. In Bangladesh, Egypt, Ghana and Zambia, the complementary 
programs are more cost effective at producing the measurable learning outcome. 13 

The complementary education models studied by the EQUIP2 team were found to be more cost- 
effective because usually the programs attained higher rates of completion and learning, much 
like the Alliance programs in Colombia. In other words they tend to be more educationally 
effective than regular public schools (as evidenced by the data on completion and learning 
presented in the previous section). For example, all the models except for Honduras and Zambia 
have higher completion rates than those countries' public schools. For cases in which data is 
available to show student learning for both public schools and complementary models, the models 
either outperform or match the public schools on the same measure of learning - often by a lot, 
and always while serving significantly more disadvantage students and doing so with less qualified 
teachers (DeStefano et al., 2006). 

As the comparison shows, the Alliance programs are achieving similar results to other programs 
that are structured in the same way and seek to reach underserved populations in rural areas and 
the program is doing so in a more cost-effective manner. 


Economic Analysis of Alliance Programs 

According to traditional economic growth theory, growth occurs by employing more basic inputs— 
more labor, capital, or land. However, there has long been a debate in literature about the impact 
of education on technology use and production. Those in favor argue that education augments the 
knowledge and skills of the workforce, which helps them to adopt better production techniques 
leading to increased productivity. In 1996, Behrman, Birdsall, and Kaplan conducted an empirical 
study on the quality of schooling and labor market outcomes. These authors discovered that 
school quality was an important factor in labor market outcomes and that investments in school 


13 In Bangladesh, the measure of learning is the primary end of cycle competency exam. In Egypt and Mali, 
student pass rates on the primary certification examination are used. In Ghana, data are available from a 
minimum competency test administered to School for Life students, and those data are compared to 
national CRT pass rates for public schools. In Zambia, community school and public school student learning is 
measured by a single same minimum competency exam that all students take. 
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quality and quantity were particularly important. The authors further concluded that there were 
likely to be substantial productivity-equity tradeoffs in schooling investments due to the 
productivity gains from focusing financial resources on improving both the quality and quantity of 
schooling. 

Making schooling more relevant and keeping kids in school can improve completers' future 
earnings potential. Lockheed et al. (1980) summarized the findings of 18 studies related to the 
effects of education on the productivity of small agricultural landholdings in 13 countries. The 
authors concluded that in most cases, education had a positive effect. They demonstrated that for 
every four years of completed formal schooling there was an associated 7.4% increase in the 
productivity of the landholdings. 14 Barros and Mendonca (1995) examined the effects of education 
on profits in the rural regions. They concluded that the productivity of a worker is reflected in his 
skill level. In their analysis of factors that impact income inequalities in Brazil, they concluded that 
education or skill level was a primary determinant. Those with higher skill levels were usually part 
of a more modern sector and were more productive. 

The impact of education can be measured in the following ways: (a) increasing access to 
education; (b) increasing completion rates so students obtain higher levels of education; and (c) 
increasing the relevance of education so students can face their immediate challenges. The 
Alliance programs in the Caldas Region have long supported investments in education with the 
hopes of having an economic impact on the coffee-growing region. Previous sections have already 
shown how the Alliance programs have increased access, completion and learning in a cost- 
effective manner. The following analysis looks at the economic impact of the Alliance programs on 
coffee production in Caldas between 1997 and 2010 based on data provided by the CGC. 

Table 7 shows that the level of education of coffee growers in the Caldas region has increased 
since 1997. The percentage of coffee growers with a primary education increased from 72% to 
77% during the 13-year period. From 1997 to 2010, the percentage of coffee growers who 
completed secondary education increased by 10 percentage points 15 (8% to 18%). It is also worth 
noting that density 16 of the coffee growing increased to an average of 5,112 17 in 2010 compared to 
4,055 in 1997. As density increased, the age of the trees has decreased. This finding seems to 
indicate that coffee growers are using techniques to increase density per hectare and focusing on 
improvements to make the crops more productive (i.e. shading, ground preparation and disease 
control). 


14 One word of caution should be mentioned. It is important to note that each of the regions examined by this study 
have their own distinct characteristics so this percentage may vary among the countries studied. 

15 The 2010 data was only a sample of 1500 coffee growers compared to the 1997 census data, which represented 
nearly 40,000 coffee growers. Therefore, it is likely that these increases may be underestimated. It is also important to 
note that 1500 represents less than 5% of the total population of coffee growers in Caldas, so the AED team cannot 
make generalizations about these findings to the larger population. 

16 Density is measured by the number of trees per hectare. 

17 The ideal density for coffee growing is between 5,000 and 10,000. 
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Table 7: Coffee Growing in Caldas 


Factor 

1997 

2010 

Average Age of Coffee Grower 

49 

51 

Gender 

79% Male 

84% Male 

Level of Education 



Primary 

72% 

77% 

Secondary 

8% 

18% 

University 

3% 

3% 

None 

13% 

0% 

Area for Coffee Growing (ha) 

2.0 hectare 

3.0 hectare 

Age of Coffee Trees 

11 years 

7 years 

Density (# trees per hectare) 

4,055 

5,112 


Source: Original tabulation created by authors 


To look at whether the increases in education level could be attributed to the Alliance programs, 
the AED team disaggregated that data by age group as shown in Table 8. Given the 13-year time 
frame, those under the age of 30 would have been entering the secondary level in 1997. Findings 
show that this age group was more likely to have completed secondary education compared to 
coffee growers over the age of 31. In fact, the percentage of coffee growers completing secondary 
education during this 13 year period increased by 21 percentage points (13% to 34%) compared to 
the 9 percentage point increase seen among older coffee growers 18 (8% to 17%). This increase 
coincides with the implementation of the Alliance programs including Posprimaria (1988), Escuela 
y Cafe (1996), and Escuela Virtual (1998). In fact, more than 140,000 students have enrolled in 
grades 6-11 since 1997. Since the Alliance is responsible for expanding education in the rural areas 
of Caldas, this increase in education level is likely to be attributed to their programs. 

Table 8: Increase in Schooling by Age 


Level of Education 

% of Coffee Growers 
under age 30 with 
Education 

% of Coffee Growers 
over age 30 with 
Education 

1997 

2010 

1997 

2010 

Primary Education 

77% 

63% 

72% 

78% 

Secondary Education 

13% 

34% 

8% 

17% 

University 

2% 

4% 

3% 

2.5% 


Source: Original tabulation created by authors 


While the expansion of access to higher levels of education is important to economic growth, it is 
also important to examine whether increased education impacted economic production and 
growth in the region. Using density as a proxy for production/productivity, the AED team 
disaggregated the coffee growers' data by level of education to see if higher levels of education 
impacted density. As Figure 5 below demonstrates, the higher the level of education, the more 


18 The increase in secondary education among older coffee growers could be attributed to participation in the 
Posprimaria program as well as the CGC, government and NGO supported adult education programs. 
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likely coffee growers were to be in the density range of 5,000 - 10,000 (ideal density). From 1997 
to 2010, density increased from 3,965 to 4,976 among coffee growers with a primary education. 
For those who acquired a secondary education, density increased from 4,591 to 5,366 on average. 
Similar to the findings above, when the data was disaggregated by age groups, coffee growers 
under the age of 30 were more likely to have a density over 5,000 and have younger trees than 
their older counterparts. 


Figure 6. Density and Level of Education (1997-2010) 
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Source: Original figure created by authors 


To perform a more rigorous look at the relationship between education and production in Caldas, 
a regression analysis was conducted and the variables of education level were correlated with 
density (as a proxy for productivity). These strong correlations provide compelling evidence as to 
the impact of education on coffee growing in the region. In both 1997 and 2010, the results of the 
correlations were positive - education was correlated with increased density in the production of 
coffee at a statistically significant level. 19 In 1997, each additional level (i.e. primary, secondary, 
and university) of education was correlated with a 15.6% increase in density. What this indicates is 
that for every additional level that a coffee grower completed, the density of coffee planting also 
increased by 15.6%. 


In 2010, each additional level of education was correlated with a 25% increase in density and was 
also statistically significant. Correlations further showed that coffee growers with more education 
were more likely to have larger plots of land and younger coffee trees, suggesting that they are 
using newer production techniques to increase productivity. Gender was not statistically 
significant, and the age of the coffee grower was negatively correlated with density - meaning 
that the older the coffee grower, the lower the density of production. Older coffee growers also 
tended to have less education. 


19 

The result was significant at the .000 level indicating that there is less than a 0.001 chance that the result occurred by 
chance. 
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Using the 2010 data set of 1500 coffee growers provided to AED by CGC, and assuming that each 
coffee tree produces an average of 2 lbs of coffee annually (1.1 kg), the AED team found that 
coffee growers with a secondary level of education or higher produced an average of 
US$14, 674/farm annually compared to US$12, 625/farm annually for those with only primary 
education, based on the average density of production for those in this category. If every coffee 
grower in the data set had at least a secondary education and produced an average of 5,366/trees 
per hectare (average for the data set), then the increased income to the Caldas region would be 
US$2.5 million/year - or an 11% increase annually in coffee production. This analysis shows that 
increasing the education level of coffee growers contributes to the economic growth of the region. 

Another way of understanding the economic impact of education is to look at the income gained 
through the provision of increased education from 1997-2010. In 1997, the average coffee grower 
had a primary level of education; had an average of 2 ha of land, and produced at a density of 
4,055 trees/ha. Based on these averages, his annual income would be US$22,305. If one assumes 
no changes in any of the above factors over the 13-year period (i.e. assumes that education did 
not improve, but stayed constant), the total economic contribution of the coffee growers would 
be an average of US$12.3 billion over the 13-year period. If one now assumes that over 13 years, 
the education level of the coffee growers increased along with density (i.e. 77% have primary 
education; 18% have secondary education and 3% have University education; they have an 
average of 3 ha of land and produce an average of 5,623 kg/ha), the economic contribution from 
these growers would be US$17.2 billion, or nearly US$5 billion more than had the coffee growers 
maintained the same level of education. 

Finally, this study also used the enrollment data in Table 9 to examine the economic contribution 
of students who have enrolled in the Alliance programs. The table summarizes a sensitivity 
analysis, by estimating the economic contribution that would be made to the economy of Caldas if 
higher percentages of students chose to enter and remain in coffee production. First, using data 
from 2010, the AED team examined the economic impact if 18%, 20%, 30%, and 50% of the 741 
grade 11 students entered coffee production. 20 If 18% of the 741 students enter coffee 
production, as a group, their economic contribution to the region is approximately US$5,624,968. 
This contribution more than doubles if 50% (or 371 of the 741 students) were to enter coffee 
production. If 50% of today's grade 11 students were to enter coffee production, their combined 
economic contribution would be US$15,624,911. 

In addition to estimating the potential contributions of Grade 11 students, we estimated the 
contributions from all students who have enrolled in secondary education (grades 9-11) over the 
life of the program. Using the same percentages for the sensitivity analysis, the AED team found 
that if 5,968 of the 33,158 students who have enrolled in the Alliance programs entered coffee 
production, their potential combined economic contribution would be approximately 
US$251,704,036 annually. If 50% of the students who enrolled in grades 9-11 entered coffee 
production, their potential economic contribution on the Caldas region would be an estimated 


20 We chose 18% as a starting point because currently 18% of coffee growers in Caldas have completed 
secondary education. 
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US$699,177,878 annually. Ensuring that greater numbers of students complete the Alliance 
programs and enter coffee production could increase the annual economic contribution of Caldas 
by a quarter to half a billion US dollars. 

Table 9: Economic contribution of the Alliance Students Entering Coffee Production 


Number of Students 
Entering Coffee 
Production 
2010 

Production 

(kg/ha) 

Annual Productive contribution 

Scenario 1: Grade 11 students entering Coffee Production (2010) 

133 (18%) 

5623 

$ 

5,624,968 

148 (20%) 

5623 

$ 

6,249,964 

222 (30%) 

5623 

$ 

9,374,947 

371 (50%) 

5623 

$ 

15,624,911 

Grade 9-11 students entering coffee production 

5,968 (18%) 

5623 

$ 

251,704,036 

6,632 (20%) 

5623 

$ 

279,671,151 

9,947 (30%) 

5623 

$ 

419,506,727 

16,579 (50%) 

5623 

$ 

699,177,878 


Source: Original Table created by authors 


While this analysis fails to take into consideration variables such as environmental factors, 
changing international prices, and inflation, it does show that coffee growers with a higher level of 
education have higher levels of production. The findings also show that there are economic 
incentives for the Alliance program to ensure that an increasing numbers of students go into and 
stay in coffee production. The finding is supported by the econometric analysis that shows that 
education is correlated to higher density, which is a proxy variable for productivity. As the 
literature has also demonstrated, the relevance of education is important and students who 
graduate from the Alliance programs learn important techniques related to coffee growing that 
help increase productivity. It is likely that the increased density seen among the younger coffee 
growers in the 2010 data sets is related to the implementation of these techniques. These 
programs are supporting the local economy, but they are also teaching students about important 
skills such as leadership and civic engagement. The following section presents the baseline results 
for grade 9 students and their leadership skills. 


Social Contribution: Leadership and Democratic Characteristics of 
Alliance Students 

Leadership 

"Leadership is nothing but the quality which makes a person stands out different from other 
ordinary people. It is associated with a person who has aggressiveness in speech and action, love 
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for their team, and who can handle pressure under different circumstances and a person who is 
always ready to fight for the rights of others. A leader is useless without followers. It is the 
followers who make a person as a leader and if required overthrow him." 

As part of the survey, students in the Alliance programs were asked to define leadership. As the 
quotes at the beginning of this section demonstrate, students did have differing definitions of 
leadership; however, more than 90% of students defined leadership as, "Someone who leads a 
group." A few students defined the word to include characteristics such as working effectively in 
teams; someone who can communicate effectively; or someone who is visionary. As the Alliance 
programs move forward, it is critical that students expand their understanding of the term 
leadership and leader. The following quotes represent definitions provided to evaluators by the 
grade 9 students in the Alliance programs. 

"Leadership is a person who is responsible for leading a group and charged with guiding the 
activities that the group feels is important" 

"Leadership is about the person who leads a group" 

"Leadership is the ability that a person possesses to guide and orient others" 

"Leadership is the ability a person possesses to lead and organize a group or team" 

Each of the definitions is true and accurate. While many definitions exist, there are a few common 
qualities of leaders that the literature supports. These qualities include: 

1. Ability to communicate in an effective manner 

2. Honesty 

3. Visionary outlook 

4. Ability to select and work with a team 

5. Ability to Listen - Action speak louder than words 

6. Ability to motivate people 

7. Consistency 

8. Ability to stand up to critics 

Students were also asked to provide examples of how they demonstrate leadership in the Alliance 
programs. While there were more than 100 different types of examples provided, the most 
common included: 

1. Participation in student government 

2. Working in teams 

3. Voting for student government representatives 

4. Leading different activities in the school 

5. Being the leader of a group 

To baseline student perceptions of these leadership skills, the AED team surveyed students from 
the Alliance schools in grade 9 and asked them to rate themselves in terms of their capacity in 
these various areas of leadership. The AED team then conducted a 360° assessment by asking 
teachers and peers to also rate the students on the same skills. This process allowed the team to 
triangulate the data and ground student's self perceptions in those of their peers and colleagues. 
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This process of conducting 360° evaluations is common in most leadership development programs 
because it allows comparing personal perceptions against the perceptions of others. 

In all, 269 students in grade 9, 255 peers and 29 teachers completed the survey. In terms of the 
student population, 159 students were women and 110 were male with an average age of 15. For 
the teacher population, 19 were female and 10 were male. Table 10 below summarizes the 
average responses for each question and group. 

Overall, students believed that their strongest skill was listening actively to others and hearing 
their words and feelings (64%). This perception was supported by the teachers, 72% of whom also 
felt that students in grade 9 were strong listeners. Among the students, 69% of the female 
students believed that they had good listening skills compared to 56% of the male students. In 
comparison, only 56% of the students' peers believed that their colleagues had good listening 
skills. 
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Table 10: Summary of Student, Peer and Teacher Survey Responses 


Question 

Student Self Rating 

Peer Rating 

Teacher Rating 


% 

Average 

rating 

% 

Average 

rating 

% 

Average 

rating 

Listens actively to those with whom s/he 
works; Hears their words and their 
feelings 

63.6 

Well 

55.5 

Well 

72.4 

Well 

Maintains an open, warm relationship 
encouraging others with praise and 
genuine respect of their views and 
feelings. 

55.8 

Well 

55.1 

Well 

51.7 

Well 

Question 

Student Self Rating 

Peer Rating 

Teacher Rating 

Provides clear feedback, reinforcing positive 
contributions, clarifying and confronting as is 
helpful. 

53.2 

Well 

49.6 

Well 

58.6 

Well 

Elicits information and ideas by asking open- 
ended questions. 

44.6 

Well 

47.6 

Well 

51.7 

Well 

Mediates for others, helping them find and 
reinforce the common ground on which 
solutions can be built. 

58.7 

Well 

44.5 

Well 

51.7 

Well 

Facilitates interpersonal and group 
relationships, teaching by example and by 
making these relationships visible. 

51.7 

Well 

53.9 

Well 

44.8 

Well 

Helps groups maintain discipline and 
direction toward achievement, while 
suggesting ways all members of a group can 
participate. 

47.6 

Well 

44.5 

Well 

55.2 

Well 

Initiates ideas, actions, solutions, and 
procedures. 

56.3 

Well 

51.8 

Well 

62.1 

Well 

Elaborates on ideas, using examples and 
definitions. 

62.5 

Well 

51.0 

Well 

41.4 

Well 

Communicates ideas effectively. 

50.0 

Well 

48.2 

Well 

55.2 

Well 

Coordinates ideas, activities, and 
relationships, making sense out of the piece. 

50.7 

Well 

46.7 

Well 

55.2 

Well 

Seeks information and clarification to shed 
light on ideas and suggestions. 

55.2 

Well 

51.6 

Well 

58.6 

Well 

Provides information, bringing a base of 
understanding to the subject at hand. 

55.2 

Well 

55.9 

Well 

39.3 

Well 

Analyzes the idea, task or process, 
developing an understanding of each piece 
and its relationship to the whole. 

51.5 

Well 

55.1 

Well 

58.6 

Well 

Teaches peers to diagnose the sources of 
difficulties. 

57.5 

Well 

55.9 

Well 

58.6 

Well 

Summarizes for the group, restating progress 
and offering a decision or conclusion for 
consideration. 

45.9 

Well 

46.1 

Well 

69.0 

Well 

Evaluates progress, process and products, 
holding them up to comparison with 
standards or expectations. 

52.6 

Well 

54.5 

Well 

53.6 

Well 

Manages using a combination of 
planning, task assignment, and guidance 
to accomplish goals. 

54.3 

Well 

50.4 

Well 

58.6 

Well 


Source: Original tabulation done by the authors 
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The peer group felt that their peers were strongest in two areas: Providing information about a 
subject at hand, and teaching their colleagues to diagnose sources of difficulties (56%). The 
student self-evaluations also gave these two categories high marks (55% and 57% respectively). 
However, teachers felt that providing information about a subject was the weakest area for 
students. Only 39% of teachers felt that students did this well. 

Table 11. Five Strongest Leadership Areas, by group 


Students 

Peers 

Teachers 

1. Listens actively to 
those with whom s/he 
works; hears their 
words and their 
feelings (64%) 

1. Provides information, 
bringing a base of 
understanding to the 
subject at hand. (56%) 

1. Listens actively to those 
with whom s/he works; 
Hears their words and 
their feelings (72%) 

2. Elaborates on ideas, 
using examples and 
definitions (63%) 

2. Teaches peers to diagnose 
the sources of difficulties. 
(56%) 

2. Summarizes for the 
group, restating progress 
and offering a decision or 
conclusion for 
consideration (69%) 

3. Mediates for others, 
helping them find and 
reinforce the common 
ground on which 
solutions can be built. 
(59%) 

3. Listens actively to those 
with whom s/he works; 
hears their words and 
their feelings. (55%) 

3. Initiates ideas, actions, 
solutions, and procedures 
(62%) 

4. Teaches peers to 
diagnose the sources 
of difficulties (58%) 

4. Analyzes the idea, task or 
process, developing an 
understanding of each 
piece and its relationship 
to the whole (55%) 

4. Provides clear 
feedback, reinforcing 
positive contributions, 
clarifying and confronting 
as is helpful. (59%) 

5. Initiates ideas, actions, 
solutions, and 
procedures (56%) 

5. Evaluates progress, 
process and products, 
holding them up to 
comparison with standards 
or expectations. (54.5%) 

5. Teaches peers to 
diagnose the sources of 
difficulties (59%) 


**The percentages represent the percentage of students, peers and/or teachers who felt that this skill was done well. 

As the Table 11 shows, all three groups agreed that students generally are able to listen to their 
peers and are effective at teaching their peers to identify problems. Students and teachers also 
agreed that students in the Alliance programs are skilled at initiating ideas, actions and solutions - 
another strong characteristic of effective leaders. 

In terms of weaknesses and areas for improvement, only 44% of students felt that they were able 
to effectively solicit information and ideas through open-ended questions. Interestingly, only 38% 
of the female students noted that these were able to carry out this skill effectively compared to 
51% of the male student respondents. Twenty-seven percent of the students felt this was an area 
they needed to improve and 5% felt it was an area that needed significant improvement. The peer 
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and teacher groups also rated this category lower than other areas, but higher than the students 
themselves. 

The peer group felt that students needed to improve in the area of serving as a mediator among 
peers. Only 45% of peers felt that their colleagues were able to effectively mediate among 
themselves compared to 59% of student self perceptions and 51% of teacher perceptions. Again, 
this response had gender differences with female students believing that they were more effective 
at mediation than male students. 

Table 12. Five Weakest Leadership Areas, by Group 


Students 

Peers 

Teachers 

1. Elicits information and 
ideas by asking open- 
ended questions. (45%) 

1. Mediates for others, helping 
them find and reinforce the 
common ground on which 
solutions can be built. (45%) 

1, Provides information, 
bringing a base of 
understanding to the 
subject at hand. (39%) 

2. Summarizes for the 
group, restating 
progress and offering a 
decision or conclusion 
for consideration. (46%) 

2. FHelps groups maintain 

discipline and direction toward 
achievement, while suggesting 
ways all members of a group 
can participate. (45%) 

2, Elaborates on ideas, 
using examples and 
definitions. (41%) 

3. Communicates ideas 
effectively (50%) 

3. Summarizes for the group, 
restating progress and offering 
a decision or conclusion for 
consideration. (46%) 

3, Facilitates 
interpersonal and group 
relationships, teaching by 
example and by making 
these relationships 
visible. (45%) 

4. Coordinates ideas, 
activities, and 
relationships, making 
sense out of the piece. 
(51%) 

4. Coordinates ideas, activities, 
and relationships, making 
sense out of the piece. (47%) 

4. Elicits information and 
ideas by asking open- 
ended questions. (52%) 

5. Facilitates interpersonal 
and group relationships, 
teaching by example 
and by making these 
relationships visible. 
(52%) 

5. Elicits information and ideas by 
asking open-ended questions. 
(48%) 

5, Mediates for others, 
helping them find and 
reinforce the common 
ground on which 
solutions can be built. 
(52%) 


Source: Original tabulation completed by authors 


As Table 12 demonstrates, all three groups agree that information elicited through open-ended 
questions was an area that students need to further develop. In fact, 8% of students, 7% of peers 
and 10% of teachers felt that students needed significant improvements in this area. Female 
students generally felt they needed more improvement than their male counterparts. 

Students and peers also felt that they needed additional skill development to improve how they 
summarize for the group and how they coordinate ideas, activities and relationships - both skills 
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that are critical to effective teamwork. Students and teachers also felt that students needed to 
improve their ability to facilitate interpersonal and group relationships. 

Finally, while responses across the groups generally triangulated and were similar, there were 
significant differences in three categories: Serving as a mediator; eliciting information through 
open ended questions, and elaborating new ideas utilizing definitions and examples. In these 
three categories, individual students' self-perception was that they either did the skill well or very 
well. Their peers and teachers did not agree. Peers and teachers felt that students needed 
improvement - or significant improvement - in these three areas. 

Democratic Behavior 

Civic engagement is defined as a sense of personal responsibility individuals should feel to uphold 
their commitments and obligations as part of a community. It can take many forms, including 
individual volunteerism, organizational involvement, electoral participation, addressing issues 
within communities, and problem solving with institutions. When students in the Alliance 
programs were asked to define "civic engagement"; nearly 100% of students defined it as "a sense 
of responsibility to the community." The following quotes from students in the program highlight 
some of the definitions. 

"Civic engagement is the obligation we have to society" 

"Civic engagement is the responsibiiity we have to participate in class" 

"Civic engagement is the responsibility we have to participate in the activities of the school" 

"Civic engagement is the capacity and development of a community" 

"Civic engagement is when we commit to completing an activity in society" 

Overall students in the Alliance programs understood civic engagement and what their own role is 
within the classroom, school, and community. 

Democratic behaviors tend to be thought of primarily as associated with voting, characteristics 
such as inclusive decision-making; participatory processes; and teamwork are also elements of 
democratic behavior. Students in the Alliance programs (more than 90%) defined democratic 
behaviors as specifically related to voting for leaders. While voting is critical to democracy, it is 
important that students understand the nuances of democratic behavior and expand their 
definition and understanding of the concept to include things such as participatory decision- 
making and teamwork - ideals that are also critical to good leadership. In that sense, schools are 
natural spaces to exercise and model leadership and democratic behaviors. 

From an evaluative perspective, students in the Alliance program are demonstrating strong 
leadership skills through their involvement in activities such as leading student groups; 
participation in various committees; and supporting their communities through various activities. 
While responses to the Leadership assessment varied across the three groups, the answers 
triangulated and were similar in nature. Overall, students believed that their strongest skill was 
listening actively to others and hearing their words and feelings - a perception supported by the 
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teachers. Peer groups felt that their peers were strongest the areas of (a) Providing information 
about a subject at hand, and (b) teaching their colleagues to diagnose sources of difficulties. The 
student self-evaluations also gave these two categories high marks. Teachers, students and peers 
all agreed that students in the Alliance programs are effective at initiating ideas, actions and 
solutions - another strong characteristic of effective leaders. 

In terms of weaknesses and areas for improvement, areas included: (a) Serving as a mediator; (b) 
eliciting information through open-ended questions, and (c) elaborating new ideas utilizing 
definitions and examples. These areas are critical to effective leadership and it is important that 
students understand how to better elicit information from others as it forms the basis for not only 
making effective decisions, but also can support them in serving as a mediator. One 
recommendation for improving these areas is to encourage and support teachers to have students 
openly ask questions, but to also use the classroom as a laboratory for role play and activities that 
place students in situations in which they need to negotiate and gather information to effectively 
complete an assignment. Teachers also need to incorporate a more nuanced understanding of 
civic engagement into the curriculum so that students understand that civic engagement moves 
beyond simply voting for elected officials, but includes serving more broadly in the community. 

The following section presents the results of the schools visits done by the evaluators. 


Programs at a Glance: Student and Alumni Perceptions 

In 2010, 258 grade 9 students were asked about their participation in Alliance programs: 88% of 
grade 9 students were enrolled in the Escuela y Cafe; 77% in Escuela Virtual; and 82% in Escuela 
Alimentaria. A greater percentage of male students were enrolled in Escuela y Cafe than were 
female students. However, women made up a greater percentage of the students enrolled in 
the other two programs. 


Escuela y Cafe 

For the purposes of this evaluation, data was collected from 267 grade 9 students. Students in this 
program were asked if they had an individual student project that they were implementing for the 
program. 54% of respondents indicated that they had not initiated a project and 46% indicated 
that they had started to implement a project. On average, students who were implementing a 
project started the project in 2008. 

When students were asked what type of work they saw themselves doing in five years, 17% saw 
themselves working as a farmer/producer; 45% indicated they wanted to be an 
administrator/manager of production within the coffee-growing industry; 24% indicated they 
wanted to be a consultant or advisor within this industry; 42% indicated they would like to be a 
partner in a coffee growing business; and 23% indicated that they preferred to do something else. 


Table 13 summarizes the skills that students believe they have learned through participation in the 
Escuela y Cafe program. Overall the students felt that the program was most helpful in their 
learning how to manage fertilizers, diseases and weeds. 
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When students were asked which materials were the most useful in helping to increase their 
knowledge, 70% felt that the program guides were the most important; 14% felt that the coffee 
projects were the most useful; 5% indicated that instructor lectures were the most useful; 2% 
indicated that the internet was the most useful; and 8% of students indicated that group work was 
the most helpful. 

Table 13: Skills Learned in the Escuela y Cafe Program 


Skill Learned 

Percent Response 

Management of sprouters and nurseries 

87% 

Coffee tree planting 

58% 

Management of diseases and weeds 

87% 

Benefits of coffee 

74% 

Management of coffee residuals 

67% 

Renovation of coffee trees 

66% 

Management of progress reports and budgets 

56% 

Commercialization of coffee 

50% 

None of the above 

5% 


Source: Original tabulation completed by the authors 


When students were asked if the knowledge they gained from participation in the Escuela y Cafe 
program prepared them to manage the coffee grower farm, 10% felt that they could manage the 
entire process; 62% felt that they could manage certain aspects of the production; and 28% did 
not feel that they could effectively manage the coffee growing process. 

Escuela Virtual 

Table 14 below summarizes the 2010 student responses about the usefulness of the material they 
have learned through participation in this program. 

In terms of the most useful learning tools, students felt that the lectures and explanations 
provided by the teachers were the most useful in increasing their knowledge base (44%) followed 
by the learning guides and internet (18%). The students felt that the projects and group work were 
the least useful (9% and 7% respectively). Sixty-one percent of students felt that they had good 
computer skills while 54% of students felt that they were able to manage the internet in an 
effective way. 14% indicated that they could navigate the web extremely well, while 31% felt that 
they could use additional skills to improve their knowledge of exploring the internet. 

Table 14. Summary of Student Responses about Usefulness of Materials 


Skill Learned 

Percent Response 

Improves your performance in school 

70% 
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Helps you solve problems on your farm 

31% 

Improves your relationship with others 

56% 

Consult new information 

76% 

Register information 

50% 

Other 

3% 

None of the above 

0% 


Source: Original tabulation completed by the authors 


Students felt that overall, the Escuela Virtual assisted them in consulting and gathering new 
information. They felt the program was least helpful in teaching them how to solve specific farm 
problems. 

Overall, student perceptions about the Alliance programs are similar to the perceptions the alumni 
had of the programs. The AED team interviewed a group of alumni during a focus group session. 
When alumni were asked about the programs that represented the most practical value to them, 
the interviewees agreed that the Escuela y Cafe, and the Escuela Virtual programs were 
instrumental in their search for employment. The Escuela y Cafe certifications provided by SENA 
were highly valued by the interviewees. Some of the youth expressed concern because, since the 
program, was not begun until 2007, alumni who completed earlier were not beneficiaries of the 
program. Twelve of the interviewees mentioned that they have a strong connection to the 
agriculture field, and that they would like to continue working in this field. Fifteen interviewees 
noted that the self-learning guides served as the best way to learn and retain knowledge, followed 
then by the teacher explanations provided. 

The same 15 respondents explained that they worked on individual student projects at home. Five 
of them mentioned that they are still working on the project programs initiated while they were at 
school; these projects have the support of their parents. Two others mentioned that they are 
taking additional certifications provided by SENA for projects such as avocado cultivation, and 
another boy is studying diesel mechanics. Ten alumni want to pursue entrepreneurial activities 
and business administration. Flowever, they were concerned because employment opportunities 
were scarce in their municipalities. When they were asked about their willingness to migrate to 
other Departments or cities, they responded that they would like to stay close to their farms and 
create local enterprises for their community. Seven of the interviewees mentioned the Conexion 
Laboral program run by CGC as an opportunity to find suitable jobs based on the competencies 
they acquired. 

Conexion Laboral is a relatively new program created by CGC in 2009 with the intention of serving 
as an employment solution for alumni of the Alliance Program, and for local employers who need 
qualified people. This is a new program that is developing job seeking, career management, 
recruitment, entrepreneurism and talent management products and services. According to the 
Conexion Laboral managers, this Program has a database of 931 students and alumni registered. 

Of those, 267 have been certified by SENA on the Escuela y Cafe competencies and 15 of them 
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were certified as trainers and evaluators enabling them to certify other alumni. This is a very 
promising program that ensures that graduates from the Alliance programs continue to work in 
Caldas, thereby improving their lives and life in their communities. This program would certainly 
benefit from an institutionalized monitoring and evaluation system that tracks student hiring. 
What's more, the program would benefit from tools, products and databases such as those 
developed for job seekers (Monster career mapping) and social entrepreneurship organizations 
(Ashoka). In the near future, the program could explore alliances with SENA and other 
organizations that provide English and other languages certifications and links with COLFUTURO or 
other programs that offer courses overseas. 

Escuela y Seguridad Alimentaria 

In 2010, when students were asked whether they had developed and implemented an individual 
student project under this program, 55% of students indicated that they had not developed a 
project while 45% indicated they had developed a project. Fifty-seven percent of students felt that 
they had gained good skills to develop a project while 7% felt that they had excellent skills to do 
the same. Thirty-two percent of students felt that they had satisfactory skills and an additional 3% 
indicated they needed significant help to develop a project. 

When students were asked which of the learning tools was most effective, 63% felt that the 
learning guides served as the best knowledge base, followed by explanations the teacher provided 
(23%). Similar to the other programs, students felt that the projects and group work were the least 
useful mechanism for learning (8% and 4% respectively). Sixty-nine percent of students felt that 
this program had helped them and their families learn a lot about better nutrition, while an 
additional 30% felt they had learned at least something about better nutrition. 

According to CGC, more than 1,000 students are currently engaged in getting their Program 
certifications from SENA. Seventy-seven percent of students indicated that they had not begun 
producing food on their farms since participating in the program - an answer resulting in large 
part because most of the students were already producing some food on the farms prior to their 
beginning the program. Fifty-two percent of students indicated that they already produced food 
on the farm prior to participating in this program, but that since their ESA experience they had 
diversified their production; fifty-nine percent of students indicated that they already produced 
food prior to their ESA participation, and have not diversified production since joining the 
program. 


A View inside the Alliance's Schools 

Schools represent a vital element in any successful effort to improve the quality of learning. This 
section of the report will contribute to understanding the school's role by examining the main 
issues and inputs shaping the Alliance's schools: school resources and conditions, teaching 
conditions, school climate, community involvement and sustainability. The issues included in this 
report were raised by those interviewed in the 30 sample schools. 
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The data collection teams interviewed a total of 94 teachers and school principals using the CIT. 
This method of data collection asks education stakeholders to focus on one or more critical 
incidents that they experience as a way of grounding the interview in concrete examples. The tool 
enabled the AED team to better understand the interplay of factors and to identify critical 
incidents that stakeholders believe to have moved the Alliance's programs in certain directions 
over the 25 years. The CIT also helped the AED team to recognize the confluence of factors that 
has led to the success of the Alliance's programs. 

These researchers observed that in the aggregate, average students in these schools seemed 
active, participative, not shy, possessed high self-esteem, and expressed themselves well. 

Teachers were generally well-engaged with students, proud of their schools and their work, and 
some seemed to have strong links with the community. For example, one school in the 
Pensilvania, allowed the community use of the school premises for community activities, such as 
meetings, sport activities, and celebrations. Another school had volunteer programs where 
students assisted the community. One of their programs assisted the community's elderly with 
their specific needs by assigning each elderly person to a particular small group of students. 

Teachers were notably actively engaged with students, and proud of their schools; many appeared 
to have strong links to the community. For example, one school in the Pensilvania allowed the 
community to use school premises for community activities such as meetings, sport activities, and 
celebrations. Another school had volunteer programs in which student participants were 
encouraged by teachers to assist the community to implement activities. One program assisted 
the community's elderly with their specific needs, assigning an elderly person to a particular small 
group of students. In sum, the perceptions of the researchers at these 30 schools were that 
students and teachers appear active, engaged, and linked to their respective communities. 

Schools Resources and Conditions 

In 2003, the MOE reorganized and consolidated schools nationwide. Colombian schools are now 
organized into three main categories: instituciones educativas (large schools) that offer grades 1 to 
11; centros educativos (medium schools) that offer grades 1 to 9; and escuelas (small schools) that 
offer grades 1 to 5. 

The public rural schools in Caldas that were interviewed for this report have an average size of 
approximately 130 students (it is possible that when examining the SED database for this factor 
alone, results at the department level could vary slightly). Most of the rural schools are located far 
away from the municipalities making education provisions difficult to secure. 

Basic Equipment and Resources 

Once a school decides to participate in Alliance programs, they receive basic equipment and 
resources from the Alliance. Basic equipment and resources include: school libraries with reading 
books for primary schools; and school libraries with specific book titles for the Escuela y Cafe, 
Escuela y Seguridad AHmentaria, and the Escuela Virtual programs. Additionally, schools receive 
furniture such as trapezoid tables, chairs, kitchen utensils for the school cafeteria, materials for 
physics and chemistry laboratories, and tools for the development of the individual student 
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projects under Escuela y Cafe and Escuela y Seguridad Alimentaria. The equipment and resources 
compliment the self-learning guides and modules for the core subject areas. The AED team 
observed that in all of the classrooms visited there were blackboards, sufficient space for seating, 
and learning corners. 

For most of the teachers and school principals interviewed, the basic equipment and resources 
received from the Alliance were identified as critical to engage students in learning. They noted 
that though the CGC had provided most of the resources originally, more recently the Alliance had 
been providing most of the resources. A few interviewees mentioned the equipment was not 
sufficient and the physics and chemistry laboratories were not fully functioning because of the lack 
of essential tools. 

Physical infrastructure emerged as a major ongoing need for schools. Fourteen teachers and 
principals expressed concern that classroom space is in such short supply that some students have 
to take classes in hallways. Others reported that their schools do not use the physics and 
chemistry labs due to the lack of proper space for the equipment. Many interviewees recognized 
that the school renovations that were made available years ago were carried out by the CGC's 
support, but that most of the current support received by the Alliance for all sources is not in 
infrastructure but in other areas such as training, pedagogical support, and materials. Interviewees 
value these contributions by the Alliance, and suggested that additional support in remodeling and 
basic physical infrastructure was needed. Ensuring that the schools receive these resources as well 
as the sufficient teaching guides and lab equipment will ensure that the Alliance schools continue 
to perform better on learning assessments and contribute to productivity growth in the region 
because students will be able to apply what they are taught in classrooms. 

Student Self-learning Guides and Teacher manuals 

Self-learning guides and manuals refer to the textbooks used in the classroom to teach all core 
subject areas following the methodology of Escuela Nueva. The Escuela y Cafe, Escuela y Seguridad 
Alimentaria and the Escuela Virtual programs have other modules and self-learning guides 
developed for the specific needs of each of these programs for secondary education (Posprimaria 
and Educacion Media). These self-learning guides and manuals for secondary are one of the 
unique features of the Alliance programs on a national scale. Some teachers interviewed 
mentioned that, when these materials were designed, the Alliance organized work sessions with 
teachers in the microcentros, which they found to be advantageous. The Alliance's guides have 
also been revised and updated with new components such as the GLG for grades 10 and 11. 

The Escuela Nueva self-learning guides for primary education were developed by the MOE for 
rural area schools that use the "modelos de educacion flexible,” or flexible education portfolio, 
including Escuela Nueva, Telesecundaria, and Aceleracion del Aprendizaje, among others 
nationwide. The MOE has recently developed new versions of these guides and they will be 
printed and distributed to the schools in the next several months. 
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The self-learning guides are perceived as very important, especially for teaching the Escuela Nueva 
methodology, as they lead teachers and students through the pedagogical practices that should be 
taught in schools. As one teacher noted: 

"Of course, these help facilitate (referring to the guides) the work in the rural area because one 
can have two, three students in one course working on the guides... this sustains the work in 
the rural areas..." 

"... The guides are excellently designed; they include tools to track student progress for each of 
the subject matters. They also include the main competencies that the students should achieve; 
the [guides] promote group work and creativity..." 

"...The guides [act as] a collaborator for the teacher because our work is based on them, so the 
guides are created to support us..." 

This data is supported by the leadership surveys that showed that students in the Escuela y Cafe 
and Escuela y Seguridad Alimentaria believed that they gained the most knowledge from the 
learning guides. 

Fourteen teachers reported that Posprimaria and Educacion Media schools received the guides on 
time and received replacements when they were modified. In contrast, interviewees felt that the 
guides for primary school were outdated and not distributed as frequently as was needed by the 
schools. Twenty-eight teachers and principals expressed concern about the primary school guides 
because the guides in the schools were old, used, and out-of-date. The guides for primary school 
had not been modified for a long time by MEN. As a consequence, some teachers have been 
developing their own guides with content that is more relevant to students. Examples of concern 
voiced by teachers and principals regarding the guides for primary grades follow: 

"... We asked the school authorities ' Where are the material (guides),' and they told us, ‘We 
were waiting for some guides, but they didn't arrive.' Teachers have to perform magic. The 
teachers go and develop their own guides, pay for the guides themselves, and distribute 
photocopies (of their self-made guides). But it would be good if we had the real guides, those 
made by the Alliance." 

"...One guide for 20 students just does not work..." 

"...In the case of the guides, those for primary are so obsolete, the teacher has to make a 
guide..." 

"...For a long time we haven't had guides. ..they are old and outdated. Many teachers have 
even made their own guides because the real ones are obsolete..." 

One issue that emerged from the interviews was the view of many teachers that since the Escuela 
Nueva guides are self-study guides, active teaching on their part is unnecessary. When examples 
of ineffective teaching were offered, 20 teachers identified that coming to class and distributing 
the guides to students, and then leaving the classroom while the students were expected to work 
alone was not a very effective practice. Another misuse of the guides was identified: when 
students had questions or doubts, some teachers simply referred them back to the guides. This 
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behavior suggests that some teachers misunderstand how to implement the Escuela Nueva 
methodology. While the students should work either alone or in groups with the guides, the role 
of the teacher is vital when students have questions or concerns. The Escuela Nueva method calls 
for teachers to rotate through the classroom, assisting individual students or working with them in 
small groups. 

In such cases as teachers simply give the guides to students without any assistance or guidance, 
they are inadequately implementing the Escuela Nueva methodology. The guides must be 
complemented by other materials and activities; many teachers reported having to adapt the 
guides and to expand upon many of the subjects broached in the guides for their particular 
students and classroom situations. 

The following quotes are examples of what teachers and principals reported as being inadequate 
practices for student learning: 

"Since these guides are called "self-study" guides, teachers assume they must be for working 
alone. It is a bad practice; a bad use of the methodology. That's how I see it." 

"...there are some who go (to class), distribute (the guides), and many times leave the students 
alone. The students are left with no guidance..." 

"Students say "Teacher, explain this to me." And the teacher will say "No, read there, by 
yourself." 

As previously stated, the learning guides were considered the most important source of "learning" 
by both teachers and students. These guides ensure that teachers have guidance in presenting 
materials and developing appropriate learning activities. As the Alliance programs move forward, 
staff need to work closely with the MOE to ensure that these guides reach schools as quickly as 
possible. Ensuring that these guides reach the schools and are used in an interactive manner is 
likely to further support improved innovation and learning in the classrooms, and thus continue to 
help Alliance students perform better on any learning assessments that they are expected to take 
to demonstrate learning gains. 

The Use of Computers and the Internet 

One of the most frequently quoted best practices among the interviewees was the use of 
computers for learning within the Escuela Virtual program. Twenty-eight teachers and principals 
noted that they appreciated the fact that their schools had a computer lab. As one teacher noted: 

"...Now we do not have to teach about computers by showing a picture of one; now we can 
actually touch them and use them..." 

Access to the internet and software provided by the Alliance was another frequently mentioned 
example of how students learn better. Interviewees stated that computers grab student attention, 
ultimately making learning easier. Many teachers admitted pride for how quickly students have 
learned the use of various software programs and especially the internet. The use of the internet 
has opened a new window to the outside world and positively affects how students in rural areas 
conduct research. This finding contrast to the perceptions of the students that participated on the 
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survey, presented in the previous section. Some of the quotes that relate to these examples 
include the following: 

"The boys and girls love (the computers), and they have very good concentration when they 
are in front of the computer. " 

"... When they go to the computer lab to do an activity on any subject, student interest is held, 
they become more involved in the activity, and they use more technological tools. This 
motivates students to learn." 

"I believe that the activities in the computer lab are interesting for the students and their 
attention is held by this..." 

"...If you are going to, for example, teach the vowels, they aren't just there in the classroom, 
but also on the computer. So, it is useful. I work with the program, and the students themselves 
are learning to use the program..." 

Some examples offered during interviews of activities implemented on the computers that are 
seen as effective learning tools include the following: 

■ reading activities 

■ research on the internet 

■ e-mail 

■ writing in Microsoft Word, Story Book, or "Mi Pequeno Escritorio" 

■ physics and chemistry in a program called “Cocodrilo" 

■ making movies with the "Crayola" program 

■ activities or adaptations described in the guides 

■ English lessons with a beginner's program 

■ making presentations in Microsoft PowerPoint 

■ taking virtual courses with SENA 

Having the Escuela Virtual program in rural schools with educational software programs and 
internet access is seen as an advantageous tool that is used regularly within the schools. The labs 
are of great interest to children and prepare students for life after graduation. According to one 
teacher: 

"...(Students) have a lot of opportunities (after learning to use a computer). The student learns 
a lot. Once a student leaves here, he goes to the city, and he is not going to be afraid when he 
sees a computer. Neither is he going to say that he doesn't know how to operate a computer." 

Twenty-one teachers were thankful to have a computer lab and internet access in the classroom 
within the Escuela Virtual. As noted, these resources come from different sources, with most 
computers being donated by Compartel (a public-private alliance that provides computers and 
internet connectivity), the local government, and in a few instances, the Alliance. Software is 
mainly provided by the Alliance. The benefits of computer labs are many, when available and fully 
functional. However, many teachers and principals mentioned that the number of computers they 
had acquired was not sufficient for the number of students in the school and that some of the 
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computers were not working properly. One teacher described the computer situation in her 
school: 


"...There is a deficiency of computers and we have two or three machines that do not work. 

They lack a hard drive. There are about four of five computers that work for a student body of 

more than 120 students." 

Other frequently mentioned challenges schools face with computers were internet connectivity, 
electricity outages, and a lack of funds to pay the electric bill. Due to the computer labs, many 
schools encounter high electric bills. In the past, teachers have had to contribute their own money 
or raise money to pay the electric bill. To illustrate these limitations, the following quotes were 
given by teachers and principals who have faced these dilemmas: 

"...There are two; three professors who pay for the cost of internet service themselves..." 

"...We have some computers, and teachers and students took time to be trained to use them. 

They go to the computers, and they don't turn on. The donated equipment is obsolete..." 

"...We had around four computers, but no internet..." 

"The internet is very restricted because 24-hour internet access just started this week. 

However, the computers don't work today because they are being worked on..." 

Investments in the computer labs are a critical component of the Alliance programs and staff 
should continue to ensure that every school has an effective computer lab with access to the 
internet. Similar to the PROJOVEM program in the state of Sao Paulo, Brazil, computers and 
internet access provide students with a rich source of learning materials specific to agricultural 
development. In schools where books and literature on agricultural production may be scarce, the 
internet can provide students with access to the most current techniques, best fertilizers, and 
access to organizations that can provide guidance to improving production. The Alliance programs 
need to continue to invest and ensure that schools have computers and access to the internet as it 
significantly contributes to student learning beyond resources in the classroom. 

Another program that supports the use of computers and internet is the Virtual Offices 
(consultores virtuales). This program is an Internet-based site created by CGC to provide further 
support to the students and families of the Alliance programs. This is a basic and intuitive site that 
creates a community of practice between students and a group of experts that exchange 
questions and answers around different topics and specific needs relevant to the rural 
communities in Caldas. Students and teachers post questions on different topics, including the 
curriculum knowledge areas as well as topics on computers, graphic and web design, agricultural 
cultivation, human development, self-help, entrepreneurism, etc. Experts hired by the Alliance 
respond to these questions. The site creates a database that captures and organizes all the 
questions and answers by topic that can later be accessed by the Alliance community members. 

The AED evaluation team had access to the site and was able to read some of the exchanges 
between the students, experts, and teachers. This program was not included within the scope of 
work of the evaluation, therefore was not assessed. However, the AED team considered it 
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important to share a qualitative observation. The virtual offices program appears to be a well 
thought out program with a lot of potential to expand learning beyond the walls of the classroom. 
Currently, social media is an important component that is being tested in United States and other 
countries as a pedagogical tool for learning. Similar experiences from the business world are also 
being tested to allow employees to create communities of practice around management, thereby 
reducing the costs of face-to-face meetings. Familiarizing students and teachers with social media 
as a pedagogical tool could position CGC at the front lines of this endeavor. In addition, the 
Alliance might benefit from linking the virtual offices with the conexion laboral program, and with 
the padrinos and microcentros, connecting teachers to other teachers and the padrinos on a 
regular basis, thereby complementing their face-to-face meetings. The Alliance is well positioned 
to expand the partnerships that it already has with Microsoft, internet providers and other 
communications companies, as well as to explore new partnerships with other social media 
companies in Colombia and overseas. It would be ideal if future evaluations, collected program 
information on the most common questions listed, as well as the areas of the virtual offices most 
consulted by members. This serves as a mapping exercise that allows the Alliance to do internal 
formative evaluations that improve program support. This mapping exercise could easily be fed 
into the Alliance information system. 


Teaching Conditions 
Teacher Training 

Teachers are the key agents in making education work - operationalizing policy and curricular 
goals while taking into account the local realities of the rural communities and classrooms, as well 
as the potential and limitations of each of the students. Thus, teacher capacity and motivation are 
critical factors for well functioning education systems. The AED team explored two main practices: 
teacher training and "on-the-job" support for teachers provided by facilitators called padrinos. 

The Alliance's programs prepare teachers to teach the Escuela Nueva methodology in primary and 
secondary education by using student-centered teaching practices. The Alliance schedules on- 
going training sessions throughout the year that cover all programs. Teacher training was 
mentioned by 22 interviewers as a critical element for teacher development. 

Twenty-one interviewees mentioned that teacher training is the main way to strengthening the 
teaching and learning practices in the classroom. A total of 22 teachers mentioned that they would 
like to receive more training. Fifteen teachers and principals viewed this training as a tool needed 
to implement all of the Alliance programs and the Escuela Nueva methodology. As some teachers 
noted: 

"(Training) has been a big support from the Alliance. Those constant training sessions in each 
one of the projects, such as Escuela Virtual, Seguridad y Alimentaria, Media— all these have 
strengthened the processes so that things keep getting stronger and (the Alliance should) 
continue with the training support." 

"...immediately after new teachers arrive, maybe a month later, they attend training..." 
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"...the training is excellent and it occurs every year. At this moment they [Alliance] already 
requested the names of the teachers that arrived and are new..." 

Twenty-one teachers interviewed acknowledged that without the training provided by the 
Alliance, even using the learning guides that are part of each methodology, teaching would be very 
difficult. The complexity of the Escuela Nueva techniques that involve overseeing and managing 
student learning at various paces and studying different sections of the learning material at one 
time is seen as difficult to manage in a classroom setting without the guidance given during 
training sessions. 

Eighteen teachers recognized that to understand and implement the methodology of Escuela 
Nueva it is necessary to either take the Alliance training or be mentored by a more experienced 
teacher; a teacher cannot simply pick up this capability on his or her own. Similarly, teachers noted 
that most of the games they know and apply in the classroom are learned in training: 

"(referring to the implementation of various math games). ..everything has been done by the 
Alliance through the training that they give..." 

Twenty-eight interviewees perceive training to focus on using learning games, pedagogical 
techniques, software, and the methodology of Escuela Nueva as a way of diversifying their 
pedagogical practices and as a way of bringing new ideas and activities into the classroom that 
support student learning. However, as noted above, most teachers also believe that in order to 
make these new practices sustainable and effective, more training should be provided. 

Additionally, training for the software used on Escuela Virtual computers is considered essential 
since most teachers become computer literate for the first time themselves in these training 
sessions. This training is provided by Padrinos of Escuela Virtual during school visits, for both 
students and teachers, or in scheduled training sessions also given by the Alliance in Caldas. 

Teacher training was seen as a critical element in implementing the Alliance programs. However, 
some interviewees were concerned that training has not been offered regularly for teachers that 
are in the small schools, new teachers hired in May 2010, and the content of the training was not 
aligned with the preparation for student tests. Four teachers mentioned that those teachers who 
do not work in the central schools do not receive as much training as those who do. Five teachers 
reported at the time of the interview that they had not received Escuela Nueva methodology 
training, despite having been in the education system for more than a few months. 

A total of 18 interviewees voiced concern that the current training was not aligned with some of 
the standardized tests, such as ICFES and SABER, which are now used to evaluate schools. These 
interviewees mentioned that they would like to be trained on the development of evaluations 
similar to these standardized tests, as demonstrated in the following quote: 

"...train teachers on the development of testing materials, in that kind of testing, because they 
(teachers) do what they think is correct and these days you see that the ICFES tests have 
changed a lot— now you can crash with tests like this one..." 
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Twenty-one interviewees identify that the teacher-training component could be sustainable in 
that it could be passed down from one generation of teachers to the next. However, they pointed 
out that it would be important that teachers who have completed the Escuela Nueva training 
devote time to the implementation of the full methodology in order to guarantee that new 
teachers are adequately trained. 

The Escuela Nueva teacher networks are a critical element to make sure that teachers obtain 
relevant training based on their own needs. The network functions by being intimately tied to 
each school's microcentro. The microcentros, represented by all the teachers in one school, select 
two representatives that will sit at the Junta Municipal de Red (municipal network). There are 29 
Juntas in Caldas that meet three times per year: One meeting to plan teacher training activities, to 
generate school innovations, and to develop an action plan with adequate resources. Another 
meeting takes place to follow-up the implementation of these plans, and the last meeting is 
organized at the end of the year to present results to school communities in a dia de logros. The 
networks in that sense are examples of how teachers' voices and needs are transformed into 
action plans that look to improve the Escuela Nueva model. Each year the Juntas prepared 
together one action plan for the Department of Caldas. This action plan is supported by the 
Alliance and their padrinos and other local stakeholders. Implementing this plan is done in 
collaboration with the SED, Alliance, Municipalities and other local actors, each of them providing 
resources to achieve the training plan objectives. This plan has become an active component of 
the education policies of the administration running the Department. As the CGC explained "the 
network is a human network, where teachers meet other teachers to work together, and to learn 
from feedback. The Escuela Nueva teacher network is the voice of the teachers to make changes in 
education policy." From an evaluation perspective, having one coherent teacher training plan that 
echoes the voices of local teachers, and thereby fosters innovative activities that improve teaching 
and learning at the classroom level in rural schools, results in an effective strategy to ensure that 
teachers get the support they requested, in the areas they identify as most in need. From an 
institutional perspective, having one concerted training plan devised by local teachers, facilitates 
streamlining the support needed by the SED and the Alliance to actually implement the plan 
correctly. 

Overall, the teacher-training program is a strong contribution by Alliance Staff and provides a 
greater degree of learning for teachers than the training they receive from the MOE. Teachers felt 
that the training seminars directly strengthened their skill set and they believed that without the 
training provided by the Alliance, it would be difficult to implement the Escuela Nueva 
methodology. The majority of teachers felt that the training has helped them to implement 
innovative activities and different types of pedagogical practices in the classroom. While the 
teachers had a lot of positive things to say about the training, concerns included: (a) the need for 
additional training in smaller schools; and (b) more alignment with preparing students for 
standardized testing. One recommendation is for the Alliance to consider a more demand driven 
training. In several AED programs, teacher professional development has begun to move in this 
direction. Under a more demand driven system, staff in a program conduct a needs assessment to 
identify the specific skill areas in which teachers need the most assistance. The training program is 
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then developed as a menu of subjects and areas base on the needs assessment. Smaller 
workshops and seminars are held throughout the year and teachers select and register for the 
areas they believe they need the most additional training in. The workshops are smaller, more 
focused, and more diverse, therefore addressing more of the direct instructional needs of 
teachers. 

School Visits by Facilitators 

The Alliance created the Plan Padrino in 1996, with teachers coming from the Colonia Escolar la 
Enea, and other teachers identified as best teachers by their peers. This team of teachers grew by 
including professionals and technicians in other areas such as agronomists and technicians in 
coffee production. The following abstract describes how interviewees understand the purpose of 
these visits: 

"The padrinos, hired by the Alliance, are specialists in areas such as Escuela Nueva, Escuela 
Virtual, Escuela y Cafe, and Escuela de Seguridad y Alimentaria. Their main purpose is to 
support schools in the process of implementing each one of the programs. There are padrinos 
for each one of the Alliance programs, and they visit schools between two and four times per 
year. Therefore, schools receive several visits from different specialists throughout the year, 
depending on the number of Alliance programs each school is implementing. Their visits last an 
entire school day, and the purpose is to advise teachers, train teacher on specific topics, 
implement activities with students, review where the school is in terms of pedagogical 
processes, follow up on the commitments the school set during the last visit, and agree on new 
commitments for the next visit." 

A total of 29 teachers and schools principals reported that these technical visits were essential to 
support their pedagogical processes. They said that these visits help teachers to learn about their 
strengths, their weaknesses, and new teaching practices. Interviewees also stated that these visits 
motivate students and teachers, and kept them on track with the processes were supposed to 
follow. As many interviewers mentioned: 

"...(School visits) are very good, they are very appropriate. They leave us reports where they tell 
us our challenges or strengths. ..we have had excellent padrinos. ..However, they have not 
benefited us as much as we'd like because we would prefer visits every month or every eight 
days..." 

"(Padrino visits)" are excellent and help us to strengthen support for program 
implementation..." 

"...we have done well in the support visits that the Alliance performs. Every year there are three 
to four visits and they support the implementation processes of Escuela y Cafe, Seguridad 
Alimentaria, Escuela Virtual, and the methodology of the Posprimaria Escuela Nueva..." 

According to teachers interviewed, one of the additional advantages of these school visits is that 
teachers and principals receive a report of the visit after facilitator has left the school, which 
includes a description of the visit, agreements made, and next steps for the teacher to follow-up. A 
copy of the report is also sent to the SED. This is a valuable accountability tool. 
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Although interviewees valued the school visits, five voiced concerns over the frequent turnover 
among the padrinos. The replacement of some of them has affected the follow-up on 
commitments and processes at the schools. Also, a few teachers commented that the padrinos 
should visit those schools that are located further from the central schools, because they felt that 
the padrinos visits were mainly in the central, easier to reach schools. 

School support visits are one of the most important elements to successful schools. Research 
conducted by the EQUIP2 team in 2006 found that what complementary education programs 
lacked in resources for compensating teachers, the programs invested in resources devoted to 
providing an extensive on-the-ground network of teacher and school support and supervision 
(DeStefano et al., 2006). In both Afghanistan programs, and in Egypt, Ghana, and Mali, schools are 
visited at least once a month by teacher support staff that observe instruction and provide 
immediate, on-the-spot feedback and professional development. In Bangladesh, BRAC program 
officers visit schools as frequently as twice a week. In Colombia, teachers who participated in this 
evaluation indicated that the support they received from the padrinos was critical to their success. 
As with the other Complementary Education systems, the Alliance should continue to invest in this 
aspect of support as it clearly brings significant value added to the classroom and additional 
support and training should translate into improved student learning. 

School Climate 

The values, attitudes, expectations and behaviors more or less shared by the members of a school 
community are often referred to as school climate. A positive school climate places an important 
role to foster student learning. Using the CIT teachers and school principals were asked to describe 
incidents that foster the school climate. 

Student Behavior in the Classroom 

The student conduct in the classroom was considered as a positive consequence of the proper 
implementation of Escuela Nueva methodology. Twenty-four teachers and principals agreed that if 
Escueta Nueva is implemented correctly, learning inside the classroom is student-centered. This 
means that the student has an active role in the learning process. 

Inside the classroom, for example, students can take on different roles such as being the table 
leader or materials keeper. The students are also in charge of their own learning since they are 
allowed to advance at their own pace. For instance, in a Posprimaria classroom there is a table for 
each grade, in this case 6 th , 7 th , 8 th , and 9 th grades. At every table, each student could be learning 
different material by following the various unit guides for the different grade levels. This means 
that for students who are quick learners, it is not necessary to wait for the rest of the class to 
advance. For those who need more time to learn, there is no pressure to move along with the rest 
of the group. 

Being in charge of their learning allows students to advance through the material at their own 
pace, but also allows for flexible promotion and attendance. If a student needs to be absent during 
coffee harvesting, he can catch up with his grade or be promoted upon finishing all units of a 
specific grade once he returns to his studies. 
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"This has always been a program implemented in rural areas, to facilitate student studying at 
harvest time. They advance at their own rate of learning, and it is not necessary that they 
complete the academic year as it is laid out January to December. Students can graduate to 
the next grade level before or after the official academic year, depending on their own 
progress. Also, students can be absent during the coffee harvest..." 

The Escuela Nueva methodology contributes to improve the relationship between teacher and 
student, the teacher becoming an active mentor. Students learn to work in small groups, with 
specified roles and responsibilities. Some teachers and principals agree: 

"The student is more responsible for his role (in the learning process). The student has a self- 
learning guide, which tells the student which activities he is going to do. I notice that students 
are very good at reading and comprehending the activity, and understanding how to get 
started." 

"...[students] relate better with their classmates, they work in groups, and they are more 
interested in other activities. This helps even more." 

Both teachers and principals noted that students learn better when they perform activities in 
which they are encouraged to be active, participate in learning games, or simply play. Forty-one 
teachers suggested that positive pedagogical practices were those in which students were active, 
entertained, and in which learning was achieved. Some examples given include the following: 

■ teaching Math with games 

■ teaching geometry with puzzles; multiplication with games 

■ using bowling pins to add and subtract 

■ learning new vocabulary with the game Scrabble 

■ learning the English words for body parts by touching them and saying the words at the 
same time 

■ using computer programs such as Story Book to write stories using games like Bingo to 
quiz students 

Four different teachers shared their perceptions of hands-on activities: 

"Games are a way to work with students more didactically, where students acquire the 
knowledge more easily. Students like that type of learning because is not so monotonous..." 

"Students learn more easily by playing." 

"(regarding how to teach fractions). ..the teacher creates games to facilitate understanding of 
that information and put it into practice." 

"What seems most innovative to me is when boys and girls learn didactically, with games and 
activities. They learn more easily this way and don't focus solely on what the guides say, since 
it is necessary to make adaptations. This keeps what the kids do from becoming too 
monotonous." 

Ten of the teachers and school principals interviewed responded that the student individual 
projects, considered "hands-on" activities, are sustainable because communities have learned to 
grow their own food to obtain the benefits of diversifying their diets and reducing malnutrition. 
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Additionally, according to the interviewees, many students have started their own student 
individual projects on their families' farms, in which their families take over ownership of them 
and continue the projects without school supervision. Student projects that involve raising 
chickens and eggs were also considered sustainable since school graduates often have the facilities 
on their own farms for these activities, and these are chores that are often assigned to them 
anyway. 

All of these projects also produce their own revenues, which can be reinvested and to make 
sustainability a reality. However, the perceptions of the teachers and school principals regarding 
this issue contrast with the perceptions of the students that participated in the survey, presented 
in the previous section, and with the perceptions of the parents that participated in the focus 
groups. 

During interviews, teachers and principals were asked for observations of what changes had 
occurred at the schools since the implementation of the Alliance's programs. The question was 
asked in a very broad way so that interviewees could point out whatever changes they had seen in 
the school environment that had resulted from the various components of the programs. 

Twenty-four interviewees responded that the new attitudes and behaviors observed from 
students were most noteworthy. In the past, students had been simple receivers of information in 
the school environment. They were not active participants in their own education, but rather 
receivers of the work and efforts performed by teachers and school administrators. This reflected 
in their general attitudes, and many interviewees reported that students were generally more 
timid, were less assertive, and doubted themselves and their work. 

After implementing the Escuela Nueva model, interviewees mentioned that students act with 
more responsibility. Since they are responsible for following guides and progressing through study 
material independently, they have taken a more active role in their own education. Teachers 
noted that this increased level of responsibility had made students more responsible overall. 

Interviewees also noticed that students had become better communicators, since they spend 
more time communicating in classrooms - a finding supported by the leadership and democratic 
survey given to teachers, students and peers as part of this evaluation. This finding results from 
the increased number of interactions among students during group activities, with some taking on 
leadership roles. It is also a result of students giving presentations in class, something that was less 
common before the Alliance's programs. Students are also perceived to have learned to 
collaborate with others better. This result from activities involving group work and activities that 
forces students to participate actively rather than passively. Teachers noted an increased 
willingness to work together and share ideas. This was also seen in the participation of students in 
the student governments. 

Some relevant quotes from the interviews included: 
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"The student is more responsible for his role. The student has a guide, some direction from the 
professor who tells him what activities are going to be done, and I see that the students are 
very skilled at reading about and understanding the activities and getting started with them..." 

"The changes, regarding the students, are in their participation, in the role of the student. The 
role changed a lot because they started to be responsible for taking the initiative, organizing 
student government..." 

"...now the boys and girls lead. ..you would be really surprised (by their initiative). ..they say to 
you, "Teacher, let's do such and such. Why don't we send a letter to the principal to see if he 
gives us permission to do such and such", or "Tell the principal that we can do this or that..."" 

"...the kids are no longer shy, now they are not timid if somebody (an adult) approaches to 
speak with them. Before, while they wouldn't run and hide, they would at least approach with 
fear (of having to talk to an adult or authority figure). Now you can see that, for example, they 
even asked to take a photo with you (referring to the interviewer). Now they have confidence, 
before they would not have dared." 

"Well, the first change was that students from the area were concerned about their own 
education, studying to the 9th grade. This is what the Alliance implemented with the 
Posprimarias..." 

"...the students— more, more spontaneous and more participatory..." 

"...the students have developed more projects, they have ideas about creating businesses, and 
they are better at managing their projects and the student individual projects." 

These perceptions are consistent with the findings in the previous section on student leadership 
and democratic skills. 

Teaching in Multi-grade Settings 

Twenty-two teachers and principals shared a concern about the complexity of working in a multi- 
grade classroom, defined as one teacher in one classroom teaching several grade levels and sub- 
levels. 

To teach in a multi-grade classroom, the teacher needs to be guiding students not only at different 
grade levels, but also at the different sub-levels of each grade. At one table, for instance, students 
from 6th grade could be working on the same subject; however, each one of the students could be 
working on different units of that subject. If a teacher is to guide the students at this table, he or 
she needs to assist with various topics. The following are some quotes shared by teachers and 
principals regarding this trend: 

"...I am here working at various levels and when I pay attention (to one student), I many times 
have to ignore students over there. ..I'm jumping all over (the classroom)..." 

"...I'm not a magician. Imagine that you are taking care of a group of kindergartens children 
and you have to be constantly mindful of them, but you can't because you have to jump 
around to teach at six different levels..." 
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"...if you did the same thing in the city, teaching at so many levels in one classroom, nobody 
would study..." 

"For example, the management of different sub-levels seems to me, well, if you don't know 
how to do it, terrible. Because you go crazy in the classroom with the various levels with 
everybody asking questions all at once, "Help me!", "No, help me!" It is terrible to work with so 
many sub-levels..." 

From the findings it is not very clear if this trend occurs more in primary, since there is one teacher 
for six grades (kinder to 5 th ), or if it also happens in the higher grades in Posprimaria and media. 
This aspect of Escuela Nueva is seen as an ineffective practice because it makes learning and 
teaching more complicated than it would be otherwise. 

Teacher turnover is a challenge to achieving sustainability of the programs provided by the 
Alliance, and a reason why teachers feel unprepared to deal with multi-level classrooms, as 20 
teachers explained. Many professionals from other sectors with no former experience in Escuela 
Nueva pedagogy are teaching in schools. When teachers lack appropriate training and support for 
teaching in a multi-grade environment they can feel overwhelmed. While the multi-grade 
approach is important in many areas and the flexible promotion and learning guide approach 
benefits many students, it is critical that the Alliance provide appropriate training for these 
teachers. As previously discussed under the teacher training section, a demand-driven approach to 
training teachers could offer specific workshops on training in multi-grade settings that is geared 
to teachers facing this specific challenge in the classroom. By having these focused workshops, 
teachers can share good practice with each other as well as learn from specialists in the area of 
multi-grade teaching. As part of the training, the facilitators or padrinos should also participate in 
the training seminars so that they also support these teachers in a more effective manner. 

Community Involvement 

Escuela Nueva includes a component that encourages community involvement and interaction 
with parents. Thirty-five interviewees explained ways in which schools interacted with the broader 
community. According to the teachers and school principals interviewed, 19 schools received 
support from members of the community, and additionally, the schools played a role in providing 
education to adults who live in the community, many of which are provided by CGC. Parents 
would sometimes agree to care for animals that schools raise as part of students' individual 
projects. During school vacations, for instance, parents have agreed to care for and feed animals 
left on school property while teachers and students are away. Parents also help at times by 
making their private land available to the schools for growing coffee or produce as part of the 
Escuela y Cafe and Escuela de Seguridad y Alimentaria programs. 

In return, some schools provide training to adults within their community at the microcentros on 
topics chosen by the community members themselves. Examples given of some of these themes 
included family planning and sexual education. Computer literacy classes are also provided, with 
school-aged students instructing participants in some cases. 
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"...the community training, induction workshops, management of the library, and work with 
the community by the teachers began in 1991..." 

"...one change has been the interaction with other schools because we didn't have this before, 
and also the support of the community. We receive a lot of support from the community..." 

"...a microcenter is part of the Escuela Nueva program. In these microcenters, everyone is 
involved; the community, the students, the teachers. It is teacher-to-teacher training, for the 
benefit of all..." 

Community involvement and engagement is a critical component that makes many 
complementary programs such as the Alliance Programs more effective (DeStefano et al., 2006). In 
the Complementary Education research conducted by the EQUIP2 team in 2006, parental and 
community involvement was found to be critical to ensuring that schools provided an opportunity 
to learn. The community ensured that the teachers were present, that students were present, and 
that the schools had the resources and support they needed to provide education to students. 
Based on findings from this evaluation, the community component contributes to helping the 
schools provide a quality learning environment through their involvement in school management 
committees and caring for resources in the schools (i.e. animals, fields). A follow-up evaluation or 
research study that looked at the extent and actual contributions of the community would 
complement this evaluation. 


Institutional Contribution: Sustainability of the Alliance Programs 

AED created a powerful tool to baseline and measure institutional changes in the education 
system in Caldas. By merging the AED-EQUIP2 systems reform framework and Beverly Anderson's 
"Continuum of Systemic Change", the AED team linked three levels of reform— political, 
institutional and technical— with the six phases of systemic change (see Annex 3). This instrument 
allows the researcher to obtain an "aerial" view of the education system from the perspectives of 
the major education stakeholders: teachers, principals, students, alumni, and SED and MOE 
officials. It also allows the Alliance to use the information for strategic planning as they look to 
continually improve the delivery of their existing education programs. 

The most significant findings presented in this study are not related so much to the magnitude of 
change, but rather to understanding how participants see the education system in terms of a 
continuum of change - and how this vision can contribute to strategic planning that can assist in 
moving education reform in Caldas forward through progressive stages. The numbers associated 
with the phases are merely intended to place the "system within a phase" and to see if it has 
progressed. Also, it is important to recognize that change is not linear, so backward movement 
through stages - or within stages— is both expected, and a pertinent signal to re-evaluate 
strategies and activities to assist in transitioning to the next phase and changing conditions. 


Overall Findings on the Systems Change Rubric 

In comparing the findings of the institutional rubric for the programs implemented by the Alliance, 
the 96 teachers, school principals, parents, and SED staff interviewed indicated that there had 
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been improvement and movement towards systems change in almost all of the categories. On a 
continuum of 1-6, with 1 representing maintenance of the old system and 6 representing 
institutionalization of the new system, there was continued movement from a state of emerging of 
new structures (5 on the scale) to actual institutionalization of the new system (6 on the scale). 
Figure 7 presents the average results across the six phases of the systems change rubric. 


Figure 7. Average results of the systems change rubric 
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Source: Original prepared by the authors. 

The aggregate results of the study indicate that the Alliance's programs have emerged into a new 
system in which the two main partners share the vision, goals, and results. In fact, the goals of the 
Alliance (access, quality, and pertinence) using the Escuela Nueva model are aligned with the rural 
education goals of the system in Caldas. As the remaining discussion in this section demonstrates, 
those interviewed for this study felt that the Alliance was transitioning into a new system that 
empowers the SED, while at the same time reduces the perception of the private supply of 
education by the CGC, creating a more mature Alliance with equal partners. Certainly, this is a 
success for the Alliance itself. 

To better understand the findings presented above and to analyze the perception of development 
among stakeholder groups, the findings are disaggregated by system element. 

Vision and Leadership 

Table 15 summarizes the perceptions of teachers, principals and SED staff on the existence of a 
coherent vision regarding the Alliance Programs. Principals reported higher scores than teachers 
in this section. However, the overall scores are not significantly different, which suggests an 
agreement of all actors on the existence of a coherent vision and leadership in the Alliance. 


Table 15. Average Vision and Leadership 



Teachers 

School principals 

SED 

Vision and leadership 

5.71 

5.83 

5.33 
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Vision and leadership refers to the political will, the clarity of vision and the extent to which 
education policy is aligned with the education goals. According to interviewees, the SED and the 
CGC have worked together for more than 28 years to promote the State's obligation to ensure 
public education in rural areas and the need to have a more qualified, new generation of coffee 
growers. The Alliance has been created between SED and CGC with a common vision and with 
goals to "improve the quality of rural education in Caldas" through the implementation of Escuela 
Nueva, Postprimaria, and other programs such as Escuela Cafe, Escuela y Seguridad Alimentaria, 
Educacion Media y Superior. This common vision is shared by the top leaders at the CGC and the 
Department of Caldas. 

The Alliance between SED and CGC is transforming the perception about public education in 
Caldas' rural areas through the enormous credibility and continuous support of the CGC and 
through each Program's results (explained above). This CGC credibility has been consistent over 
time in education and other sectors. The CGC has been the major provider of Escuela Nueva's 
learning guides, school supplies, teacher training, and physical infrastructure for rural schools in 
Caldas since the early 1980' s, as mentioned in previous sections. The CGC has provided resources 
and technical assistance in the construction of roads, aqueducts, housing, and infrastructure for 
the coffee growers' farms. As one school principal explained: 

"The CGC has always been with us; the public governors come and go. For us, the CGC is a 
symbol of transparency, while the public sector is just politigueria. The public sector trusts the 
CGC. It helps the public sector to do a better job, a job that they would otherwise be unable to 
do or people wouldn't believe in what they would be doing." 

The prestige and recognition of the CGC is a unique feature of the Alliance in Caldas and abroad. 
The CGC is recognized as a well-managed company, efficient, and effective. This credibility has 
been an example for other departments in Colombia for how to create, run, and sustain a public 
and private partnership. According to the Coordinator of the PER, the alliance established between 
SED and CGC has served other departments finding suitable private sector partners to develop 
alliances and secure financial resources distributed at the national level. 

The experience of the Alliance at the national level has led other coffee growers committees in 
other departments to explore the CGC model and to adopt similar programs such as Escuela y 
Cafe, which is implemented in the Departments of Cundinamarca, Cauca, Risaralda, Santander, 
and Tolima, with the support of CGC. 

Ninety interviewees mentioned that Escuela Nueva was the cornerstone of the Alliance, and the 
essence of the other programs. Over the years, the Alliance adopted the learning guides, training 
modules, facilitators' model, and the classroom furnishings (libraries, desks, chairs, etc) originally 
developed by the CGC. The Alliance has promoted the creation of small teachers' workgroups 
(microcenters) and school networks. As mentioned in previous sections, the small teacher's 
workgroups are ways to engage teachers in material production, refreshers of training sessions 
and to serve as a forum for them to convene around different topics (pedagogical and 
managerial). 
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For most interviewees, the Alliance has done a remarkable job extending the Escuela Nueva model 
from primary to secondary, then up to higher education, all the while maintaining the principles of 
the model. This finding from the interviews is supported by the cost and cost-effectiveness data 
presented in previous sections. 

An important piece of the Alliance is the continuous support from the CGC and its strategy to visit 
the schools. This is where the vision of the Alliance is transformed into action. As mentioned 
previously, the Alliance has a team of facilitators that visit the schools and provide technical 
support and follow up to the Alliance's main programs. These facilitators provide assistance to 
teachers in their classrooms on how to manage the self-learning guides, how to organize the 
classrooms, and specific strategies on how to work in multi- grade settings. 

These facilitators also provide support on specific topics related with the Escuela y Cafe program, 
such as the stages of coffee production, and with the Escuela y Seguidad Alimentaria program on 
how to organize the school fields, among others. As parents, teachers, and school principals 
mentioned, the facilitators are extremely visible at the school level and they are recognized by the 
school communities. The facilitators are the strongest link of the Alliance, as interviewees noted. 
One teacher commented: 

"When the facilitators visit schools they wear distinctive uniforms with the Alliance logo. This 
visual aid helps to identify the Alliance at the school level. In Caldas it is common to see CGC 
staff wearing yellow t-shirts that identify staff with the communities. Their presence is strong 
because CGC supports many areas of development including roads, infrastructure, health, 
water, and education. By having facilitators wear red vests instead of the yellow CGC t-shirts, 
local communities are able to differentiate the Alliance representatives from others." 

For local authorities, the Alliance's vision will continue. Flowever, in the future, the Alliance must 
find a way to make the model universal in Caldas to include both rural and urban areas. The 
Secretary of Education suggested that to make the model universal the SED should unify the 
materials, guides, and modules for all rural and urban schools. The SED should work with 
universities to train teachers using the Escuela Nueva model, and to work with universities, NGO's, 
and other actors to adapt the model into their programs. An important piece, he noted, would be 
to apply the model in all urban schools. Authorities believe that making the model universal in 
Caldas in the urban and rural will contribute to equitable distribution of resources, sustainability, 
teacher mobility, efficiency, and quality improvement. 

Overall, composite results indicate that in terms of vision and leadership the Alliance is in the 
emerging new structures phase (stage 5). In this stage, a clear vision of where the rural education 
system should be going is emerging, and the SED is seen as the champion of this effort. These are 
some elements of the education system that are operated in keeping with the desired new 
system. These new ways are generally accepted by the 96 interviewees. 

Many of the factors that identified the vision and leadership of the Alliance become elements that 
explain other stages of the institutional change that will be covered later. 
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Information and Data for Decision Making 

Table 16 summarizes the perceptions of teachers, principals and SED staff on the existence of an 
information system that supports decision making. The overall scores do not vary significantly, 
thereby suggesting agreement among the stakeholders. 

Table 16. Average Information and Data for Decision Making 



Teachers 

School principals 

SED 

Information and Data 

5.21 

5.12 

4.67 


The interviewees are aware of the existence of an information system at the national level called 
SIMAT that was developed by the MOE and managed by the SED. The SIMAT is an EMIS system 
that tracks coverage, repetition, student-teacher ratios, and completion rates over time. This 
system is used by the MOE to distribute national financial resources among the departments. 
Forty-two interviewees felt that the SIMAT was difficult to use and that it did not allow for 
modifications. For instance, many teachers and principals suggested that SIMAT does not account 
for "flexible promotion", a unique feature of the Escuela Nueva model that respects a student's 
individual pace instead of resorting to a pass or fail system. This limitation of SIMAT is an 
institutional barrier to further progress, and affects the teacher, the student, and the school at the 
end of the year. 

Teachers noted that at the end of the year they felt forced to make a "subjective" assessment of 
the student to make him or her pass or fail. School principals felt pressured to pass the students, 
even if they were not ready for the next grade. The students that were not ready to pass to the 
next grade will likely retrocede because they don't have the basic skills needed for the new grade. 
The SED team believes that a lot of progress has been made around information for decision 
making in the Department of Caldas. 

In 2007, the SED was certified in quality standards 21 and is now in the process of strengthening 
municipal secretaries to improve their governance and management structures, including 
improvement in financial procedures, human resources, and quality. The SED is also developing 
the SIGCE system and is integrating different sources of information. Flowever, for the SED 
interviewees, while the SIMAT has a defined purpose at the national level, the system is not well 
known within the Department of Caldas. What is known are the statistical reports that SED 
produces every year. These reports, though useful, are not in the system and citizens are therefore 
unable to access the EMIS system to find education data or specific data for Alliance programs. 

Ninety interviewees noted that SED has a website with a hyperlink to the Alliance's website. This 
website is useful and serves as a main communication channel between the SED and the schools. 
The SED informs schools about administrative and managerial changes through the website, which 
has a "one-way" instead of a "two-way" flow of information, as information is produced mainly by 
the SED and not by the schools. 


In 2007, Colombia initiated a certification process in quality standards for each department using ISO 9001 
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School principals, who know the importance of accurate and reliable information, mentioned that 
it would be useful to have data available at the school level to compare schools in quality and 
efficiency. When asked about the Alliance's programs, interviewees responded that they would 
like to have access to information regarding students, teachers, inputs, and outcomes. They would 
also prefer to have comparative information about the student individual projects that students 
are developing in Escuela y Seguridad Alimentaria, Escueta y Cafe, and Escuela Virtual. They want 
to be able to upload the students' certifications, issued by the SENA, into the system for those in 
the Escuela and Cafe program, the Conexion Laboral program that tracks alumni, and the robotics 
program developed through Escuela Virtual. Many interviewees suggested having all the 
achievements, or dia de logros, listed on the Alliance's website. 

Interviewees further noted that even though teacher rotation occurred frequently, there is no 
system to track how many times a teacher had received training for the Alliance's programs. The 
information system should track all training activities implemented through the Programs. 
Teachers interviewed felt that the information system should also include a record of the activities 
performed by the microcentros, the teacher networks, and other programs including the 
production of teaching materials. Parents interviewed felt that more information was needed to 
filter down to the school level. They wanted to know if their children are learning as well as other 
children were in different schools. Many suggested that their children in the rural areas learn more 
that their urban counterparts but they do not have the data to support this view. Parents would 
like to have more information and many suggested that they would like to learn Escuela Virtual so 
they could access information from the SED in real time, instead of waiting for the school principal, 
or the local mayor to visit the school. 

Eighty-nine interviewees felt that at this stage of the institutional rubric more improvements 
needed to be made. The composite scores show that the Alliance and the SED information system 
is in the Transitioning phase. In this phase the stakeholders are aware of the importance of 
information to improve decision-making. Interviewees recognize the need for, and are willing to 
take concrete actions to, transition to a system in which information flows from the SED to the 
users and vice-versa. 

Institutional Capacity 

Table 17 summarizes the perceptions of teachers, principals and SED staff on the availability of 
resources to support the education sector. The scores vary somewhat between SED staff and 
principals. The school principals perceive that schools have all the resources needed for the 
programs to run, while SED staff members believe that they are in a transitioning phase. 

Table 17. Average Institutional Capacity 



Teachers 

School principals 

SED 

Institutional Capacity 

5.57 

6.00 

5.00 


Institutional capacity refers to the human and resource capacity of organizations to implement 
and sustain reform efforts. In the case of the Alliance, institutional capacity means having trained 
leaders who demonstrate the necessary skills to develop and roll-out the Escuela Nueva, 
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Postprimaria, Escuela y Cafe, Escuela y Seguridad Alimentaria and Educacion Media y Superior 
programs as well as sustain program initiatives. The sustainability of interventions depends on the 
institutional status of the Alliance. 

The Alliance's main contribution to institutional capacity is its human resources: the leadership of 
the coordinators and program managers, facilitators, and teachers. The CGC managers have been 
with the Alliance since the onset, which has insured continuity, sustainability, and a concrete 
implementation of the Alliance's vision and goals. 

As mentioned previously, one of the strongest links of the Alliance is the constant support at the 
school level by professionals trained by CGC, called facilitators or "padrinos." These teachers, with 
extensive expertise in Escuela Nueva, are crucial to providing technical support to other teachers 
in Caldas' rural schools. Teachers at the classroom level are contributing to the development of 
Alliance programs by producing guides, or modules, that complement the materials used in the 
programs. The same teachers work together in the microcentros and the teacher networks, which 
are spaces where teacher meet to discuss pedagogical and management issues. The microcentros 
and the networks empower the teachers and allow them to connect with other teachers as well as 
with the SED and CGC. Along with microcentros, there are the multipliers, or teachers that are 
identified to serve as facilitators. A detailed examination of how these components interact would 
be important for future research. 

The interviewees, composed of 36 teachers and school principals, mentioned that because of high 
teacher turnover, the Alliance's programs were suffering and schools did not have trained 
teachers to support students, especially at the beginning of the year. Many believe that teacher 
training was one of the most compelling components of the Alliance but recognize that it is 
difficult to provide continuous training due to high costs. Other interviewees thought that since 
teacher turnover was an issue at the national level and beyond the area of action of the Alliance, 
the Alliance should focus on updating materials and guides. Interviewees also noted that the 
teacher training schools (Normales) in rural areas teach students in Escuela Nueva but lack a direct 
link with the Alliance. This link would promote future teachers to stay in rural schools and apply 
the Alliance model. 

For school principals and teachers, as mentioned above, the guides for primary education for 
Escuela Nueva are outdated and are not relevant for all students. Teachers noted that they have 
worked with the MOE to update the guides to respond to new national standards for the core 
subject areas. However, the process has been long and is still not complete. 

Few interviewees linked institutional capacity with specific programs of the Alliance. Some noted 
that Escuela Virtual could be strengthened if the Alliance worked with computer service providers 
to improve their services. Many computers are damaged, the internet fails, and few teachers are 
trained to properly use the equipment. As one mother commented: 

"I understand that this issue is complex and that the school cannot solve the problem with the 
computers that are damaged. However, it is not fair that my son is in ninth grade and does not 
know how to turn on the computer, which is not sufficient anymore. He needs to find 
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information, save information, send it... they need to have an email address to communicate 
with the world. If we don't have computers we will fall behind." 

Also, those interviewed suggested that the certification of Escuela y Cafe, during which the SENA 
evaluates a student's knowledge and skills regarding the steps to produce coffee, should be 
mandatory, and that it could be opened to other students that are not 15 years old but that are 
enrolled in the Escuela y Seguridad Alimentaria program. Currently, students older than 15 years 
old voluntarily decide if they want to obtain certification. As one teacher responded: 

"The standardized tests are important, but they are not relevant to a student's life. Let me 
provide a simple example. The CGC says that students can decide if they want to be certified or 
not in Escuela y Cafe. I am the teacher of Escuela y Cafe in my school. I teach all my students 
the same things. I prepared them as if all would be certified, even if they are not 15 years old. I 
teach them how to grow coffee from planting up to harvesting coffee. I teach them all. So, 
why can't all receive the certification? If certification is up to the child, it will not happen as a 
teenager does not understand the value of obtaining certification. Three weeks ago we 
finished, and those who are 15 years old will receive the certification. Those who will not get it 
this year are prepared to be certified when they turn 15. The Alliance is giving us modules, 
training, inputs, technical assistance; I cannot allow my students to not benefit from this 
support. In my class, my policy is that every student will get certified. The teacher of Escuela y 
Seguridad Alimentaria is doing the same." 

Students in the Escuela Seguridad y Alimentaria are eligible for certifications given by SENA. 
However, the current law states that students under the age of 15 cannot be certified. While this 
law exists and is important, perhaps the Alliance could work with policy makers to adapt the law 
allowing younger students to be certified as long as they were not used for the purposes of child 
labor. It is an issue that should be carefully explored so that students can receive a credential of 
some sort, but ensure they are protected from leaving school and entering the workforce too 
soon. 

In one school, the CGC Program called Mis Costos (tracking my farm's expenses), a CGC program 
for coffee growers to manage coffee production using planning and financial tools, was adapted 
for the Escuela Virtual and Escuela y Cafe programs. For the teacher of this program, Mis Costos is 
an example of institutional capacity applied beyond the classroom. For her, this is an example of 
how learning is pertinent to the realities of a coffee grower's life. 

The composite scores show that the institutional capacity is moving from the emerging new 
structures to institutionalization of the new system. 

Institutional Framework 

Table 18 summarizes the perceptions of teachers, principals and SED staff on the existence of an 
institutional framework that supports program implementation. Stakeholders noted that they are 
in a transitioning phase. The overall scores do not vary significantly. 
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Table 18. Average Institutional Framework 



Teachers 

School principals 

SED 

Institutional Framework 

4.85 

5.68 

5.33 


The institutional framework is the legal structure around which education functions. It involves the 
legal statutes and charters that guide the rules of the game, as well as the implementing 
regulations and rules. These rules, regulations, and charters can both help and hinder reform 
efforts that are being institutionalized. Successfully identifying which legal statutes can and should 
be changed provides opportunities for real reform to be sustainable. 

The Alliance has been institutionalized under the management structure and legal grounds of the 
Secretary of Education. The programs are included within the development plans, the 
departmental budget, and the accountability required by law. The Departmental Development 
Plan has a chapter about rural education that includes the main activities offered by the Alliance, 
in which the indicators for success are the same indicators of the Alliance. There is an annual 
agreement that is signed between the two parties, and approved by the Departmental Assembly, 
in which one party commits to set aside public financial resources to run the programs and the 
other party commits to executing the programs and providing additional private financial 
resources. 

The Alliance has a governing body for decision-making that includes a coordinator/supervisor from 
both the SED and CGC. This formal entity meets regularly to discuss implementation issues and to 
track progress of the programs. According to the Secretary of Education, the Alliance is regulated 
by the laws that govern the department. In this sense, the Alliance needs approval every year for 
its budget. Dealing with frequent budget approvals is a time consuming task that involves 
negotiations among the governor, the finance and planning secretaries, and the CGC. 

Political situations can make it more difficult to approve the budget because every four years a 
new governor arrives and the Alliance has to raise awareness and explain the programs to the new 
governor in order to receive his support in renewing the agreement. What is clear is that the 
communities buy into the programs, and the assumption is that the programs will last even if a 
new governor decides to stop supporting them. 

The interviewees agree that the Alliance is accountable to the citizens of Caldas. The Alliance has 
to report on the progress of their programs and management situation to the Governor, local 
government councils, municipal committees, local assemblies, schools, and parents. By law, the 
Alliance is required to report to the general attorney, the department controller, and other local 
accountability bodies. The Secretary of Education stated: 

"The Alliance has complied with the regulations and the requirements of the law. This means 
that we are using the resources wisely and we are efficient and effective, which is also 
important to keep good credibility with the citizens of Caldas." 

The Alliance has developed a communication strategy within the press office of the governor's 
office. This strategy utilizes departmental newspapers, a television program, radio, and the SED 
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website. The school principals interviewed noted that the SED website was the main channel of 
communication between schools and the Alliance. The SED uses the website to inform schools 
about changes in the law, training schedules for teachers, meetings with the governor and the 
SED, distribution of materials, agreements with the MOE, and many other administrative and 
managerial issues. 

The composite scores show that the Alliance and the institutional framework is in the transitioning 
phase. 

Civil Society 

Table 19 summarizes the perceptions of teachers, principals and SED staff on the role of civil 
society. The overall scores do not vary significantly, thereby suggesting agreement among 
stakeholders. 


Table 19. Average Civil Society 



Teachers 

School principals 

SED 

Civil Society 

5.00 

5.23 

5.33 


In certain cases, major opportunities for reform reside in civil society's desire for change, which 
can be pervasive throughout one country. Such support can create an enabling environment for 
reform which, when coupled with strong driving forces and political will, can affect the success of 
projects in identifying and targeting conditions for success. Civil society commonly embraces a 
diversity of spaces, actors, and institutional forms varying in their degree of formality, autonomy, 
and power. 

Civil societies are often populated by organizations such as registered charities, non-governmental 
development organizations, community groups, women's organizations, faith-based organizations, 
professional associations, trade unions, self-help groups, social movements, business associations, 
coalitions, and advocacy groups. In this definition, parents are considered pillars within civil 
society. 

Interviewees noted that the Alliance was strengthened by the individual ability of SED and CGC to 
create alliances. For the Alliance, SED and CGC have established partnerships with Universidad de 
Caldas and other local universities, SENA, Caldas's hydroelectric center (CHEC), Fundacion Luker, 
and other local NGOs. The interviewees also noted that the Alliance could strengthen the 
relationship with the local mayors. As one of the teacher suggested: 

"We need municipalities to invest more in our schools. Our microcentro organized a Municipal 
Intervention Plan that highlights the needs of the schools, teachers and students. ..we included 
some solutions. We tried to establish an alliance with the municipality to put our Plan into 
action but it was difficult. We complain a lot about our needs and sometimes forget to 
recognize all the good things that we have. This was an attempt to contribute, to act.. .but I 
guess that the municipality was not ready." 
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The school principals mentioned that they could also organized better to improve quality of 
education in the rural schools. Principals acknowledged the new initiative of the SED to create the 
School Principal Roundtable as an exercise for the Secretary of Education to meet directly with the 
school principals. They noted that this roundtable could be more beneficial if some members of 
the Alliance participated continuously. 

One of the unique features of Alliance's programs is the genuine interest by parents to participate 
in the schools. Many parents said that they were familiar with these programs because they 
visited the schools more frequently now that they were students themselves under the Adult 
Education program run by CGC with CAFAM. Other parents participate because they are alumni of 
the school, and others because, as a parent described: 

"I send my children to the school because they have a library, uniforms, computers, nice desks, 
and chairs. The Escuela y Cafe program helps them to love their land, their coffee, and their 
countryside. We live in the countryside. I want them to stay here. I want them to learn new 
techniques to grow coffee." 

Forty-one parents noted that they wanted to participate more actively in schools. They want more 
information about how their children are learning, what individual projects were being developed, 
and how much revenue from selling their chickens and coffee is invested back into the school. 

They want to know what other schools are doing so they can make comparisons. 

Many parents interviewed voiced concern about the new mandatory distribution among the 
schools by MOE. This new distribution includes one school principal in charge of six or more small 
schools. If the school principal is located in the big school, then parents tend to believe that all the 
resources are going to the bigger schools rather than the smaller ones. The distribution of the 
resources from SED is linked with the institutional capacity mentioned above. 

"The school principal does not come to visit our school. If she comes, it is once a year and she is 
always running out of time and does not have time to speak with us" 

Interviewees noted that there is a stigma about rural and urban schools. For urban teachers and 
schools principals, students in the rural areas are not well prepared, while parents think that urban 
schools are violent, insecure, and the adults do not control their students. Parents felt that having 
their children in the countryside was more secure and they had high expectations for their 
children to succeed in school. As one parent noted: 

"I want children in the countryside to stay in the countryside to grow coffee. I would like our 
children to finish eleventh grade here. I want them to go on to higher education, to the 
university, to come back as managers, experts in coffee. They should be part of the CGC staff in 
the future, instead of people from the cities that are not familiar with the farms. Our children 
will be excellent employees of the CGC because they really know the business from the bottom 
up. CGC should invest more in them; we should have access to scholarships." 

Thirty-one of the parents interviewed felt very strongly that the Educacion Media needed to be 
expanded to all rural schools and not just to the big schools. That way, students aged 15-17 will be 
placed in schools close to their homes. 
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The interviewees indicated that other local actors (small and medium businesses) participate in 
the schools by creating alliances with the SED, the CGC and the municipalities. These alliances 
included Cooprocal, Traslavega, Codemas, Cooperativa de Caficultores de Manizales, and 
Confamiliares, which are involved with the school activities and with the Consultorios Virtuales 
from the Escuela Virtual Program. Compartel, a private company is a major contributor of 
computers, internet access, and equipment to schools with the Escuela Virtual. 

The composite scores show that the civil society is in the emerging new structures phase. 
Interviewees are aware of changes in the way that the Alliance is relating to the schools. 
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Conclusions 

This evaluation sought to determine the extent to which investments in the Alliance programs led 
to economic, social, and cultural changes in the communities that the programs have served. The 
following discussion presents the conclusions arising from this evaluation. 

The Alliance has provided education access to underserved populations in Caldas where the state 
was not present. Education access data shows the success of the program in reaching these 
communities. One of the most important achievements of this Alliance has been increasing access 
to secondary education. From 2002 to 2007, access at the secondary level rose from 7,952 to 
13,074 students. This is an important achievement and one that should be shared with other 
countries. 

How cost-effective are the programs based on student learning outcomes compared to 
the national average? 

This analysis shows that the Alliance programs are more cost-effective at reaching underserved 
populations and ensuring that they survive up to grades 5 and 9 of the education system and 
learn. In terms of the cost-effectiveness of access, the Alliance programs were 4 times more cost- 
effective than the national average and nearly two times more cost-effective than the urban 
schools in Caldas. In terms of survival to grades 5, the Alliance programs were nearly two times 
more cost-effective at getting students to survive to grade than the national average and slightly 
more cost-effective than the schools in urban Caldas (US$2,049 vs. US$2,698). 

The Alliance schools were more than a US$1,000 more cost-effective than rural schools in the 
Quindfo and Risaralda regions. In terms of grade 9, the Alliance schools were nearly two times 
more cost-effective than the national average and also more cost-effective than similar coffee 
growing regions in Colombia. 

When the analysis compared the cost of obtaining learning outcomes, the Alliance programs 
continued to be significantly more cost-effective at helping students learn than the national 
average and surrounding regions. 

Based on the AED/EQUIP2 research on Complementary models of education similar to the Alliance 
schools, these schools were more cost-effective because: 

■ They create schools that are located where families live, making it easier for children, 
especially girls, to enroll in school and attend regularly. 

■ They set up community-based management structures that are able to effectively oversee 
the day-to-day operations of the schools; assuring student and teacher attendance, setting 
the calendar and schedule, collecting contributions, supporting teachers, etc. 

■ The schools develop a localized curriculum that often simplifies and focuses the national 
curriculum, supported with materials and instructional strategies that, where possible, 
relate to the local/regional context and issues. In the case of the Alliance programs, the 
programs go one step further and link school to work, by ensuring that the programs better 
prepare students to enter the coffee industry, which supports the economic development 
of the region. 
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■ Support teachers by providing training and ongoing community engagement to assure the 
success of the school 

■ Provide community-based school management committees 

These findings were supported by researcher observations, interviews, and focus groups. It is also 
important to highlight that the investments that the Alliance is making in these schools is 
contributing to better learning and economic growth in the region of Caldas - and doing so in a 
cost-effective manner. 

What is the relationship between education and productivity in the Caldas region? 

Findings from the evaluation show that education correlates to increases in density of production 
(i.e. a proxy for productivity) at a significant level. The analysis further showed that coffee growers 
with a higher level of education have higher levels of production. The relevance of education is 
also important to productivity. Interviews and focus groups confirm that the Alliance programs 
help students learn skills that they can directly apply on their farms - skills such as improving 
fertilizer use; disease control; and proper planting techniques. These are all skills that more than 
50% of students felt they were learning in the Alliance programs. 

The literature supports the fact that improvements in these types of interventions directly 
improve farm productivity. When the data was examined by disaggregating the appropriate age 
groups (i.e. those under the age of 30), results showed that these farms tended to be more 
productive. The people in these age groups were the most likely to have participated in the 
Alliance programs in the last 13 years. It is likely that the increased density seen among the 
younger coffee growers in the 2010 data sets is related to the implementation of the techniques 
taught in the Alliance schools since these schools are the only ones providing education to the 
population. 

To what extent do students demonstrate democratic and leadership skills? To what 
extent are students civically engaged? 

The ability to communicate effectively; be honest; visionary; the ability to select and work with a 
team; listen and motivate people; be consistent and stand up to critics are all characteristics of a 
good leader. Based on the 360° survey results, students in the Alliance programs are acquiring at 
least some of these skills. 

Students, peers and teachers agree that students in the Alliance program are effective listeners 
and are able to effectively work in teams. Survey results indicate that between 55-72% of 
students, peers and teachers believe that the students listen effectively. The survey results also 
indicate that students are developing strong skills related to helping their peers diagnose 
problems and developing creative ideas and solutions to problems. The surveys further show that 
many students are engaged in student government and voting behaviors at the school and 
understand the concepts and importance of civic engagement. 

Interviews with teachers, principals and community members support the findings above. 
Interviewees indicated that students served as table leaders, group leaders, and student 
government representatives. They also indicated that once the Escueta Nueva methodology was 
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implemented, teachers saw a distinct change in student attitudes that resulted in students taking 
on more responsibility for their own education and learning. The self-paced nature of the 
programs further support students taking personal leadership responsibility for their education - it 
is their responsibility to keep moving through the system. 

Finally, there is evidence to support that students in the Alliance program are civically engaged. 
Survey results indicate that they implement individual projects on their landholding; they 
participate in activities at the school and community level; and the students articulate an 
understanding of the importance of their "social commitment to the community." 

What institutional opportunities or challenges help or hinder students from succeeding 
both during and after the program? 

Overall results of the study suggest that the Alliance's programs have emerged into a new system 
in which the vision, goals, and results are common and aligned with the rural education goals of 
the system in Caldas. The programs have strong leadership and a shared vision; there appears to 
be sufficient training provided to relevant actors; and the institutional framework (legal structure) 
supports sustainability for the project. 

Students are committed to the type of learning they receive and are performing relatively well 
when compared to the peers in other parts of the country (as measured by the SABER tests). 
Education in Caldas is delivered in a cost-effective way and is linked to economic growth in the 
region. 

Currently around the world there is pressure to focus on what is happening in the schools, 
effective classroom time, school climate, teachers' techniques in the teaching-learning process, 
student and teacher expectations, acquiring reading, math, science and other skills in lower 
grades, ongoing classroom support for teachers, the use of education materials; as well as the 
introduction of tools such as ongoing assessments designed to support the teaching process, 
assure the relevance of learning, and respect the different rhythms and ways in which students 
learn. The following section offers recommendations for improving the quality of the Alliance 
programs. 


Recommendations 

Based on the results of the study and the institutional challenges faced by the Alliance programs, 
the following discussion offers recommendations for further improvements and refinements to 
the program. 

1. Since the programs are more cost-effective in terms of access, survival and learning, 

communities, donors, and others should continue to invest in these. Internationally, many 
countries are looking for solutions to the expansion of secondary education. The Alliance has 
an effective "recipe" and they should make an effort to share their experience 
internationally. 
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2. There is a positive correlation between increased levels of education and productivity. 

However, a more rigorous study of the economic impact that analyzes other important 
variables should be undertaken. 

3. Focus on creating and developing incentives for students to stay in the coffee industry 
because it contributes to the economy of Caldas. 

4. Understand the causes and effects of teacher turnover, and implement strategies to 
ensure a more stable teaching force in the Alliance programs. As indicated in the previous 
section, interviews with program participants indicated that teacher turnover is a challenge 
that affects the quality of education received by students. While some turnover may not be 
under the control of the Alliance (i.e. MOE assigns teachers to schools), the Alliance can 
conduct a study to understand why high turnover takes place and help the SED to develop 
strategies to ensure teachers stay with the program. 

5. Focus on improving the learning guides and ensuring the guides contain the most current 
information. Forming public-private alliances with publishing companies or even local 
business to cost share the development of materials for the program are feasible 
alternatives to finding the funds for guides within the existing budget. In Guatemala, the 
government initiated a campaign to ensure that every student in primary schools had books, 
notebooks and pencils. They went out to the business community and through the help if 
the private sector were able to secure donations (both in kind and financial) to meet the 
MINEDUC requirements. In Egypt, the publishing companies assisted development projects 
with the development, publishing and provision of storybooks to primary schools. These 
partnerships can be effective mechanisms for maintaining low costs and still providing new 
materials to students. 

6. Understand why students do not find the individual projects as useful as other learning 
methods. Based on similar programs that have been evaluated by members of this team, 
projects that students develop and implement tend to be seen as the most useful learning 
practice because it gives students a chance to move from theory to practice. 

As stated above, grade 9 students in the Alliance programs did not find the projects a useful 
tool and many had not implemented or even designed projects. There are several steps the 
program can take to ensure that projects are a useful tool. 

a. Conduct a research study to understand why students do not view the projects as useful 
and use the information to improve the process. 

b. Enhance students' visits to successful farms and industries so they can compare and 
contrast production strategies, practices, and tools. Have them keep journals of the 
visits and then use the information to debate with each other the pros and cons of the 
different approaches. Mis Costos program is a good practice to promote and to expand 
in other schools. 

c. Ground the projects in the student's farm. In Brazil, a similar program called PROJOVEM 
taught students to develop business plans. They had to conduct a diagnostic of the farm, 
taking into account all the resources and capital available, farming techniques that were 
used, and financial investments that were made. Based on the analysis, they developed 
projects to directly improve their own farms. They were given funds to implement the 
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projects and they repaid the "loan" into a rotating fund for future students to use. 
Enabling students to implement actual projects that increased the productivity of the 
farms gave them incentives to carry out the project. The requirement of repaying the 
loan held them accountable to their peers. It was a unique way to make projects useful 
while focusing the student on things that were a benefit to them and their families. 

7. Understand the most important data needs and collect that information. The Alliance 
programs need to understand the types of research and evaluation questions that they 
would like to continue to answer moving forward and ensure they are collecting the 
appropriate data. Relevant data based on this evaluation would include the following: 

a. Enrollment data disaggregated by gender and program; 

b. Completion data disaggregated by gender and program; 

c. Productivity and income data for the farms of students in the program; 

d. A tracking system that follows students upon completion of the program for a series of 
years and collects salary and job information for the purpose of understanding how 
students use the skills and knowledge gained from the Alliance programs. 

e. Continue to track the institutional changes and contributions of the program using the 
institutional rubric baseline. This will allow the program to monitor movement related to 
how systems change and use the information for on-going formative improvement of 
the program. 

f. Collect data related to students, teachers, inputs, and outcomes. 

g. Collect comparative information about the individual projects that students are 
developing in Escuela y Seguridad Alimentaria, Escuela y Cafe, and Escuela Virtual. 

8. Ensure the permanence of the Alliance, based on legislation if possible. The Alliance was 
formed by an agreement between the CGC and the SED, which made it effective over the 
years. However, it may be more advisable to assure its institutionality through a law 
specifying a clear and limited mandate. This law might end the "lobby" needed every four 
years to continue with the Alliance programs. The Secretary of Education mentioned in the 
interview an interesting idea of unifying the model in rural and urban areas in Caldas that 
can be further be explored. 

9. Increase public-private Support for programs. Several teachers and principals mentioned a 
need to increase partnerships with local private firms who could support learning in the 
schools. The challenge of having outside expertise was particularly true for the Escuela 
Virtual, for which teachers and principals felt that increased partnerships with local 
computer service providers would improve the quality of not only the computers but of the 
education students receive related to the uses of computers for research and work. Internet 
access and electric bills are seen as weak in some schools, which create challenges for 
students to use the technology to complement their learning. Public-Private partnerships 
with other entities can assist schools in addressing these challenges and others. 

10. Revise teacher training to include demand-driven training. As previously discussed, 
teachers and principals believe that they need to receive more training and support. 
Educators in the non-central schools feel that they are not being sufficiently supported and 
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teachers in multi-grade classrooms feel that they need more training in how to handle these 
types of classrooms. Research conducted under the EQUIP2 program shows that often 
support visits focus on management or administrative support (i.e. classroom management, 
development of lessons plans) rather than on direct instructional support (i.e. how to teach 
multiplication, how to teach phonics). 

No data was collected for this evaluation specifically about the professional support that 
teachers receive, but from interviews we infer that they need more direct instructional 
support to most effectively support students in learning skills and implementing projects at 
all grades. Multi-grade teachers need direct support to improve not only their instruction 
strategies in these classrooms, but organizational support to ensure all students are staying 
focused on learning. The type of instructional practice that happens in classrooms is critical 
to understand and there are several instruments and studies that could support the Alliance 
in better understanding teacher needs, which informs the training process. 

11. Increase support to teachers in multi-grade classrooms. Countries such as United States, 
Malawi, Egypt, El Salvador, and Equatorial Guinea have developed effective continuous 
evaluation tools for teachers to track student learning by collecting information on how 
students are progressing in their learning and using this information to make instructional 
decisions. Continuous evaluation when used by teachers' evaluation becomes an integral 
and on-going part of the process. Without continuous assessment teachers lack critical 
information and the lesson may be either too hard or too easy. When teachers have a clear 
understanding of the skills of children in their classrooms, teachers can adjust instruction to 
respond to the needs of each of the students. For disadvantaged students, continuous 
assessment presents a valuable opportunity to "get back on track" because the teacher can 
provide learning exercises that more precisely respond to the child's skills and difficulties. 
Sometimes small adjustments to the instructional process are all that is necessary to keep 
these students engaged and learning. Developing these types of tools can support teachers 
in multi-grade classrooms by facilitating the work of tracking learning among all the diverse 
ages and grades in a classroom. 

12. Improve coordination with the teacher preparation schools ( Normales ) and local 
universities. Teachers interviewed mentioned the lack of coordination between the Alliance 
training with the training on Escuela Nueva that they received in the Normales schools. By 
enhancing links of collaboration with the Normales, the Alliance could promote a unified 
student-centered teaching practice; improve new teacher's perceptions and expectations 
about students and schools to motivate them to stay in the countryside. The best teachers 
are needed in the rural areas. 

13. Enhance the role of school facilitators. School visits by the padrinos are effective and should 
continue and perhaps increase. Many countries are struggling on finding effective ways to 
provided pedagogical and effective support. The Alliance has a sound model. However, the 
visits need to focus more on the provision of instructional support (e.g. in student 
evaluation, strategies to deal with multi-grade classrooms more effectively, etc) and 
particularly in the non-central schools who are currently receiving fewer visits. In the United 
States "instructional rounds in education" are being developed inspired by the medical- 
rounds model used by physicians. Through this new form of professional learning, education 
leaders and practitioners develop a shared understanding of what high-quality instruction 
looks like and what schools need to do to support it. The Alliance could explore a pilot to 
bring together padrinos, SED teams, and local universities to develop an instructional round. 
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14. Focus on improved student communication skills. One of the most important leadership 
characteristics is effective communication. Students in grade 9 indicated that understanding 
and knowing how to effectively communicate was one of the areas students felt they most 
needed to improve. Without effective communication skills, students will be challenged in 
successfully achieving growth as leaders. Teachers should incorporate activities that allow 
students to role-play and use communication with each other as a way of improving the skill. 

15. Examine the possibility of SENA certifications for students under the age of 15 because it 
will act as an important incentive to "pull" students through the higher grades. The 

literature on school to career programs shows that certification for students can be one of 
the most important incentives for students to complete programs and continue learning. 
Several interviewees indicated that while the relevance of the education was importance, 
receiving certification from the SENA - particularly for the Escuela y Cafe would help 
students in their future work. While there is a law in place that prevents the program from 
certifying students under the age of 15, perhaps alternative certificates or credentials could 
be developed to ensure that students had the incentive to complete the programs - and 
that organizations recognized the skills that they have gained in the process of completing 
their studies. Perhaps the development of skills and capacity assessment that can be given 
to students that demonstrates their skill competency is an alternative to certification. 

16. Develop school accountability tools in which parents can participate. Several parents 
mentioned the need to get more information about their children learning, the school 
performance and the use of resources in comparison with other schools. In many countries, 
community participation is not common so country efforts focus on inviting communities to 
the schools. On the contrary, Caldas has a strong and vibrant community involved already on 
the schools. Many interviewees mentioned that parents are involved serving as members of 
the school councils, and because many are taking adult education courses at the school. In 
Central America, Peru and Namibia a model of school report card has developed to bring 
parents to collect information, monitor progress and act upon. Local entities such as the 
Observatorio de la Calidad de la Educacion Caldas are interested in joining efforts with the 
CGC to develop such tools. The Alliance is encouraged to develop these tools as a way of 
monitoring their schools. 

17. Revise the school principal's emphasis on administrative duties vis-a-vis instructional 
support. Many teachers and parents mentioned that school principals are not sufficiently 
engaged in instructional support. The new school organization increases principal's 
administrative duties by having to manage many small schools far away (even to manage up 
to 17 small schools). This is not unique to Caldas. Many countries are looking for ways to 
improve the instructional role of the school principal. The relative balance between 
administrative and instructional duties will depend on the circumstances within the schools, 
but also a matter of ministerial policy. The Alliance can guide this process in Caldas through 
management and professional development courses using the Mesa de Rectores or other 
convener spaces for principals. 

18. Conduct additional research studies that contribute to strategic planning. On-going 
research and evaluation, particularly formative evaluation always helps programs learn and 
improve. In addition to the studies already mentioned, the following additional studies are 
recommendations that come from the findings of this evaluation. 
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a. This evaluation base lined perceptions of students, peers, and teachers with regard to 
leadership and civic engagement. It would be useful to conduct a similar baseline in a 
similar region (e.g. Quindio, Risaralda) as a way of collecting comparative data that can 
be used over time to see if the perceptions of leadership are different between regions 
and if the perceptions change over time. 

b. Measure the actual skills students gain that are related to leadership, rather than 
perceptions. Observation instruments, role-play scenarios and 360° evaluations are the 
best tools for measuring capacity in areas such as team building, communication, and 
vision. 

c. Similarly, the program should conduct a baseline of teacher's pedagogical skills as a way 
of reviewing the training program and using the information to create a more demand- 
driven program that is focused on building instructional/pedagogical skill sets. In Jordan, 
a similar capacity assessment led an AED program to create a training "menu" for 
teachers based on needs and moved MOE training away from the traditional cascade 
approaches. This demand driven approach significantly increased teacher's knowledge 
and skill base over the course of 3 years. 

d. During school visits, have padrinos record what is considered good teaching practice and 
use the videos to share those practices with other teachers who are struggling with 
instruction. The videos visually demonstrate for teachers that good practice is realistic 
and helps them to see how to do it. Several AED programs use this approach to training. 

e. Develop an adaptation of the Opportunity to Learn indicators to measure if the 
conditions are in place for students to learn. OTL measures can occur at numerous 
places along the learning chain: Does the grade 4 teacher prepare adequately to teach 
the material? Does the pupil lose instructional time due to unruly behavior in the class 
or school closing due to weather? 

19. Link teacher training to Standardized tests. Several principals indicated that they are 
evaluated based on the ICFES/SABER format and guidelines but lack the training to prepare 
for the assessments. In fact, the principals noted that the Escuela Nueva training they 
receive does not align with the evaluation requirements. It is an important incentive for 
principals to do well on these evaluations so that they continue to support the Escuela 
Nueva schools. The Alliance should look at adding a component in which they teach teachers 
how to develop tests that are similar to those standardized tests students take. 

20. The SABER tests only provide one perspective on learning and do not effectively measure 
the practical learning students undertake in the Alliance programs. The Alliance needs to 
develop learning measures that are specific to the program. For example, using the projects 
as a practical measure of what is learned. 

The goals of education cannot be realized unless systems are better able to reach rural, poor 
children. The Alliance programs have succesfully reached rural students and extended education 
far beyond what was previously available and made that education relevant to their reality. 

Models such as the Alliance schools show how countries can better organize schooling in areas 
usually least served by the formal education system, and also show how different approaches to 
school organization lead ultimately to greater effectiveness - in terms of the amount of 
completion and learning the schools are able to generate as well as the type of skills students 
learn. 
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The factors that most contribute to the success of these kinds of programs include : 


■ Smaller schools established in collaboration with communities and managed by 
communities; 

■ Teachers supported through ongoing, regular supervision and training 

■ School-based decision-making and community-based management and governance 

■ Simplified curriculum and increased instructional time 

■ A curriculum that is relevant to the students future 

■ Student governance bodies that assist in developing leadership skills 

Any attempt to more broadly promote or adopt these programs needs to must consider how best 
to assure these (and any other) conditions that will increase the likelihood of success of the 
program remain in place. It is not enough to try and simply replicate the idea of community-based 
schools. 

Governments and their partners must invest the financial and institutional resources necessary to 
ensure that the conditions most favorable to success of those schools can be assembled and 
sustained. This implies drawing capacity from wherever it can best be found - asking government 
institutions to do what they do well, relying on public and private partners to do what they do 
best, and allowing communities to assume responsibility for what they can best manage. These 
are areas that the Alliance programs have built into its system and it is reflected in the findings of 
the institutional rubric. 

The Alliance is making substantial efforts to ensure educational opportunities for all children and 
youth in Caldas, but they require the decisive support of civil society and government institutions. 
As the Alliance's programs continue to move towards the last phase of systems change - the 
predominance of a new system - it is important to learn from what has already happened and to 
continue improving and conducting evaluations that inform the Alliance and the communities. 
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Annexes 


Annex 1 

Table 1. Distribution of financial resources per student from the national level to the 
departmental level in 2007 


Type Department 

Coverage preschool and primary _ , , 

, , Coverage secondary and high school 

level 

Cost per 
student 
(Colombian 
pesos) 

Minimum per capita 

no-decentralized Cost per student (Colombian pesos) 
municipalities * 

Average per capita no- 
decentralized municipalities 

i 

ATLANTICO, GUAJIRA Y QUINDIO 

919,400 

60,532 

937,000 

60,532 

2 

ANTTIOQUIA, BOLIVAR, CORDOBA, 

MAGDALENA, SUCRE, CALDAS, CESAR, 
HUILA, NORTE DE SANTANDER, 

RISARALDA, TOLIMA y VALLE 

924,600 

942,300 

3 

BOYACA, CAUCA, CUNDINAMARCA, 
NARI„ O y SANTANDER 

929,900 

947,700 

4 

CAQUETA, CHOCO META, ARAUCA y 
CASANARE 

935,100 

953,000 

5 

PUTUMAYO y SAN ANDRES 

1,306,000 

1,331,000 

6 

AMAZONAS, GUAVIAREy VICHADA 

1,776,200 

1,810,200 

7 

GUAINIA y VAUPES 

1,985,100 

2,023,100 


Source: SED, 2010 

MOE’s cost per student in secondary and high school equals to 60% of the cost of students in preschool and primary by type. 
* Municipalities that are not decentralized received US$60,532 per capita 
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Annex 2 


Calculation method of the survival rate to 5th and 9th grades for the Caldas department 
and the Alliance schools: 

The EPDC group at AED created a formula to calculate the survival rate to 5 th and 9 th grades that 
follows the methodology used by the UN Statistics Division. AED used this formula to calculate the 
survival rate to 5 th and 9 th grades for the Caldas department and the Alliance schools. The method 
of computation and formulas used are as follows: 

Survival Rates for state and coffee schools were calculated using the reconstructed cohort 
method. The method estimates the percentage of a cohort of pupils enrolled in grade 1 in a given 
school year who are expected to reach higher grades, regardless of repetition. The estimations use 
data on the number of pupils and the number of repeaters by grade for two consecutive years 
to calculate flow rates: promotion, repetition, and dropout, which are then applied in the 
calculations of survival. The method assumes that the promotion, repetition, and dropout rates 
observed in these two years for each grade remain constant over the entire period in which the 
cohort is enrolled in school; and the same rates apply to all pupils enrolled in a given grade, 
regardless of whether they previously repeated a grade. The method also assumes that drop-outs 
never return to school. 


Method of computation: "The indicator is typically estimated from data on enrollment and 
repetition by grade for two consecutive years, in a procedure called the reconstructed cohort 
method. This method makes three assumptions: drop-outs never return to school; the promotion, 
repetition and drop-out rates observed in the last two years remain constant over the entire 
period in which the cohort is enrolled in school; and the same rates apply to all pupils enrolled in a 
given grade, regardless of whether they previously repeated a grade" (UN Statistics Division 2010). 


m 
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t-1 
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Where : 


Ftj-E&i-R&i 


i = grade ( 1 ? 2 ? 3 ; t = ya ar (1,2,3,... jn), g = p upil-c abort. 

SR - j = Survival Rate of pup i- cohort g at grade i for a referent e > r e sr 1c 

J-'g = Total number of pupils belonging to a cohort g at a reference year k 
pt ph 

t.j.= Promote es from ^g who would jo in sue cessive grades i throughout succ e ssive ye srst. 


Ri = Humber of pupils rep eating grade i ia s chool-ye :±r t . 


"The calculation is made by dividing the total number of pupils belonging to a school cohort who 
reach each successive grade of the specified level of education by the number of pupils in the 
school cohort (in this case the students originally enrolled in grade 1 of primary education) and 
multiplying the result by 100" (UN Statistics Division 2010). 

"This method requires data on the number of enrollments and repeaters in each grade of primary 
education in two consecutive school years." 


Enrollment and repeaters data was provided by the government of Caldas for years 2008 and 
2009. This data was used for calculations for both the Caldas region and Alliance schools. For the 
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latter, AED filtered enrollment and repeaters data for those schools enrolled in any of the Alliance 
programs for both years 2008 and 2009. Since the Alliance focus is on the expansion of secondary 
school, the enrollment number was larger for grades 6-9 than for grades 1-5. Therefore, the 
calculation of the survival rate to 9 th grade, for the Alliance schools and Caldas, took into account 
only enrollment numbers for grades 6, 7, 8, and 9, and the number of repeaters. As shown in the 
table "Cost-Effectiveness of Alliance Programs vs. National and Departmental Public System", the 
survival rates to 5 th grade for both Caldas and Alliance are greater than 100% (122% and 120% 
respectively). One explanation is that students who dropped out may have returned to school. 
Please note that one of the limitations of the formula used to calculate the survival rates is that it 
does not take into account drop-outs who never return to school. Therefore, it was decided to use 
a rate of 100% for the survival rate to 5 th grade for both Caldas and Alliance schools. 

Basic Methodology 

For the Alliance programs, the total recurrent cost for each one of the six programs 22 is calculated, 
and then divided by the number of students to obtain the per-pupil cost of access per program. 

The per-pupil cost of access for primary and secondary at the national level was obtained from the 
UNESCO (2010) EFA report and represents costs for 2007. The per-pupil cost of access for primary 
and secondary at the Caldas departmental level is converted into USD with information provided 
by the Colombian Secretariat of Education 23 (see table in Annex 1) and represents costs for 2010. 
Recurrent costs for the Alliance programs include teacher training and salaries, materials, 
technical support, legal services, and program overhead. Development costs include the start up 
costs of programs at schools and are not included in the calculation. 

Survival rate represents the "percentage of a cohort of students who are enrolled in the first grade 
of an education cycle in a given school year and are expected to reach a specified grade, regardless 
of repetition" (UNESCO 2010, p. 449). The survival rate to grade 5 for the nation was taken from 
the UNESCO 2010 EFA report, and the rate to grade 9 was taken from the Education Policy and 
Data Center (EPDC) website, both represent a 2007 rate. The survival rates to grade 5 and 9 for the 
Caldas department and the Alliance schools were calculated by the EPDC department in AED. The 
data and formulas used to calculate these are described further in Annex 2. These rates are used 
to calculate the cost per completer which illustrates how cost-effective each one of the three 
cohorts of schools is at having those students who initially enrolled in first grade reach 5th and 9 th 
grade. 

Finally, the cost per learning outcome is calculated by dividing the cost per completer by the 
percentage of students achieving the desired outcome. In this case, the desired outcome is the 
satisfactorio and avanzado levels on the 2009 SABER tests (mathematics and reading). 


22 The Alliance programs are: Escuela Nueva, Education Media, Posprimaria, Escuela y Cafe, Escuela Virtual, and Escuela 
y Seguridad Alimentaria. 

23 Based on the - CONPES 131- (2010). 
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Annex 3 


A Systems Approach to Education Sector Reform 
The Institutional Rubric 

In the early 1990s, Beverly Anderson began working with US school districts to understand and 
research systems change in education. At the time, administrators across the country were 
recognizing the need for systems reform in education - but felt lost in the complexity of the 
process. The administrators indicated a need to understand and strategize what direction to take 
and how to make decisions. 

To provide educational stakeholders with an 'aerial' view of the shifts occurring in the education 
system, Anderson developed a matrix that looked at the 'Continuum of Systemic Change.' This 
matrix defined six developmental stages and six elements of change that represented a composite 
of experiences in implementing reform in the US education system (Anderson, 1993). 

The Stages of Systemic Change 24 

The purpose of this matrix is to provide stakeholders with a common vantage point for 
communicating, making decisions, and tracking systemic education change over time. The six 
stages characterize the shift from a traditional education system, to one that includes 
interconnectedness, active learning and shared decision-making (Anderson, 1993). While the 
matrix displays these stages as linear, it is recognized that change will take an uneven path - 
moving forward and then falling back (Anderson, 1993). The six stages of reform include: 

1. Maintenance of the Old System: Educators focus on maintaining the system as originally 
designed. They do not recognize that the system is fundamentally out of alignment with 
the conditions of today's world. New knowledge about teaching, learning, and 
organizational structures has not been incorporated into the present structure. 

2. Awareness: Multiple stakeholders become aware that the current system is not working as 
well as it should, but they are unclear about what is needed instead. 

3. Exploration: Educators and policy makers' study and visit places that are trying new 
approaches. They try new ways of teaching and managing, generally in low-risk situations. 

4. Transition: The scales begin to tip towards the new system; a critical number of opinion 
leaders and stakeholders commit them to the new system and take more risks to make 
changes in crucial places. 

5. Emergence of New Infrastructure: Some elements of the system are operated in keeping 
with the desired new system. These new ways are generally accepted. 

6. Predominance of New System: The more powerful elements of the system operate as 
defined by the new system. Key leaders begin to envision even better systems. 

The literature also indicates that education systems will move through six developmental stages. 
Monitoring these elements can assist in measuring progress. The elements described by Anderson 
(1993) include: 

1. Vision: The vision that people have of an education system and what it should do must 
change for the system to change. It is important to track the number of people and 
interest groups/stakeholders who agree on the purpose of the new system and how the 
number increases. 


24 Adapted from Anderson (1993) for NCREL 
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2. Public and political support: As the vision develops and is translated into practice, public 
and political support must also increase, which includes a deeper understanding of what 
and why changes are needed - and are articulated. 

3. Networking: Building networks that study, pilot and support the new vision of education is 
essential in establishing lasting systemic change. These networks typically do not rely on 
the existing bureaucratic structure - increased communication and information sharing is 
one way of measuring this aspect of change. 

4. Teaching and learning changes: Teaching and learning based on the best available 
research on how people learn is at the core of the new system. If changes do not occur in 
teaching and learning, then other changes have little value. 

5. Administrative Roles and Responsibilities: To achieve change in the classroom, admin R&R 
need to shift at the school, district and national levels from a hierarchical structure to one 
of support for shared decision-making. 

6. Policy Alignment: Policy needs to be aligned around the beliefs and values of the new 
system, particularly as it relates to curriculum frameworks, instructional methods, 
materials, student assessment practices, resource allocations and the inclusion of 
students. 

The purpose of the rubric is to: 

1. Develop a common language and conceptual picture of the processes and goals of change 
among diverse stakeholders. When stakeholders can see and understand the issues and 
perspectives of the others, they are better positioned to take actions that will support and 
enhance others' specific situations. A rubric such as the one used in this study can assist a 
group in formulating common language and vision of the goal toward which the group is 
moving. 

2. Develop a strategic plan for moving forward on systemic change. Systems change, once 
underway, can often feel overwhelming due to the magnitude of the tasks involved in 
moving towards change. This rubric can assist stakeholders to create smaller, more 
manageable steps towards change. An example cited by Anderson (1993) related to US 
domestic state Mathematics and Science curriculum directors who used the rubric at a 
national conference to analyze their own state's progress on systemic change. Many 
participants found that they were moving along reasonably well with the vision, policy 
realignment, and shifts in administrative roles, but that the changes in classroom teaching 
were small and public support was lagging. By using the results from the rubric, the group 
was able to refocus and develop strategies for addressing the areas needed special 
attention if the full system was to change. 

3. Use as a formative evaluation tool for ongoing assessment of the process to support and 
encourage quality change. This tool can also provide a basis for deciding where and what 
type of formative evaluation is needed to ensure change and activities are measured and 
improved throughout the process. 
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